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ABSTRACT 


As  part  of  recent  reforms  of  the  welfare  programs  in  the  U.S.,  many  states  and  localities  have  refocused 
their  Welfare-to-Work  programs  from  an  emphasis  on  human  capital  acquisition  (i.e.,  providing  basic 
education  and  vocational  training)  to  an  emphasis  on  “work-first,”  (i.e.,  moving  welfare  recipients  into 
unsubsidized  employment  as  quickly  as  possible.)  This  change  in  emphasis  has  been  motivated,  in  part,  by 
results  from  the  experimental  evaluation,  conducted  by  the  Manpower  Demonstration  Research  Corporation 
(MDRC),  of  California’s  Greater  Avenues  to  Independence  (GAIN)  programs  during  the  early  1990s.  Their 
evaluation  found  that,  compared  to  programs  in  other  counties  that  emphasized  skill  accumulation,  the  work- 
first  program  in  Riverside  County  had  larger  effects  on  employment,  earnings,  and  welfare  receipt.  In 
addition,  the  Riverside  program  was  cheaper  per  recipient  than  the  other  programs. 

This  paper  reexamines  the  GAIN  programs  from  two  complementary  perspectives.  First,  we  extend  the 
earlier  analysis  through  nine  years  post-randomization,  which  is  the  longest  follow-up  of  any  randomized 
training  program,  and  find  that  the  stronger  impacts  for  Riverside  County’s  work  first  program  tend  to  shrink, 
whereas  the  weaker  impacts  for  the  human  capital  programs  in  Alameda  and  Los  Angeles  Counties  tend  to 
remain  constant  or  even  grow  over  time.  Second,  we  develop  and  implement  methods  to  allow  the 
comparison  of  programs  implemented  by  random  assignment  in  different  places  despite  striking  differences 
in  the  composition  of  the  participant  populations.  On  a  substantive  level,  our  reexamination  of  the  GAIN 
experiment  lead  us  to  conclude  that  although  the  work  first  programs  were  more  successful  than  the  human 
capital  accumulation  programs  in  the  early  years,  this  relative  advantage  disappears  in  later  years.  On  a 
methodological  level,  our  results  suggest  that — at  least  in  this  welfare  context — these  methods  are  a 
promising  approach  both  for  the  estimation  of  program  effects  from  non-expcrimental  data  and  for 
extrapolating  program  results  from  one  location  to  a  different  location  with  a  different  population  mix. 


1.  Introduction 


The  passage  of  the  Personal  Responsibility  and  Work  Opportunity  Reconciliation  Act 
(PRWORA)  in  1996  provided  the  most  radical  reform  of  the  U.S.  cash  assistance,  welfare  sys¬ 
tem  in  the  last  60  years.  In  particular,  the  legislation  directs  states  to  reorient  their  welfare  pro¬ 
grams  toward  encouraging  earnings,  and  not  cash  assistance,  as  the  means  for  disadvantaged 
parents  to  provide  for  their  children.  To  encourage  this  “work  rather  than  welfare”  objective, 
families  under  PRWORA  are  limited  to  5  years  of  federally-funded  cash-aid,  states  are  obliged 
to  require  adult  family  members  to  engage  in  some  type  of  work  after  two  years  of  aid,  and  the 
full  funding  of  the  Temporary  Assistance  for  Needy  Families  (TANF)  block  grant  is  subject  to 
the  states’  meeting  stringent  work  participation  requirements  for  adults  in  assistance  units.  This 
combination  of  time  limits  and  participation  requirements  have  placed  increasing  pressure  on 
states  to  devise  strategies  and  programs  that  get  low-income  households  out  of  welfare  and  into 

jobs. 

The  focus  on  work  in  this  most  recent  effort  to  reform  the  U.S.  welfare  system  is  not  new. 
Beginning  in  the  1960s,  the  federal  government— through  the  Work  Incentives  (WIN)  program 
and  its  successor  program,  the  Job  Opportunity  and  Basic  Skills  (JOBS)  program  begun  in 
!  988— has  supported  employment  and  training  programs  in  an  effort  to  increase  the  employment 
rates  and  skill  levels  of  welfare  recipients.  What  has  changed  in  the  most  recent  welfare  legisla¬ 
tion  is  the  emphasis  on  getting  recipients  into  jobs  quickly  rather  than  allowing  for  the  more  de¬ 
liberate  acquisition  of  basic  and  vocational  skills  in  state  “welfare-to-work”  programs.  To 
achieve  the  federal  government’s  objective,  states  have  increasingly  relied  on  “work-first, 
“quick-job-entry,”  or  “labor-force-attachment”  (LFA)  strategies  that  aim  to  quickly  move  wel¬ 
fare  recipients  into  unsubsidized  employment  through  the  provision  of  job  search  training  and 
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assistance.  Such  programs  are  in  contrast  to  programs  that  emphasize  “human-capital  develop¬ 
ment”  (HCD),  through  more  expensive  and  longer  duration  basic  skills  and  vocational  training 
programs.1 

Although  the  movement  of  states,  as  well  as  the  federal  government,  away  from  basic 
skills  and  vocational  training  programs  and  towards  programs  that  encourage  quick  entry  into 
jobs  has  been  motivated  by  several  factors,  one  important  impetus  has  come  from  the  findings  of 
several  recent  experimental  evaluations  of  various  state  welfare-to-work  programs.  One  of  the 
earliest  and  most  influential  of  these  studies  is  the  evaluation  of  California’s  Greater  Avenues  to 
Independence  (GAIN)  program  conducted  by  the  Manpower  Demonstration  Research  Corpora¬ 
tion  (MDRC)  in  the  late  1980s  and  early  1990s.  In  this  evaluation,  welfare  recipients  in  six  Cali¬ 
fornia  counties  were  randomly  assigned  to  either  a  “treatment”  group  that  was  to  receive  services 
in  a  county  based  and  designed  “welfare-to-work”  program,  or  to  a  “control”  group  to  which 
these  services  were  denied.  Counties  were  given  considerable  discretion  in  the  types  of  recipients 
they  selected,  as  well  as  in  the  way  they  designed  their  programs.  Thus,  in  effect,  the  MDRC 
study  was  an  evaluation  of  six  separate  programs,  each  with  its  own  distinct  population  and  each 
with  a  within-site  random-assignment  design. 

The  largest  effects  on  participants  were  found  for  Riverside  County’s  GAIN  program. 
Among  female  heads  on  AFDC  at  the  time  of  their  enrollment  in  GAIN,  Riverside’s  program 
boosted  annual  employment  rates  and  earnings  by  39%  and  63%,  respectively,  over  the  first 
three  years  after  randomization  compared  to  those  of  non-participants  and  reduced  annual 
AFDC/TANF  participation  rates  by  8%.2  In  contrast  to  the  programs  run  in  the  other  analysis 

1  See  Friedlander,  Greenberg,  and  Robins  (1995)  and  LaLonde  (1997)  for  more  on  government-sponsored  training 
programs  in  the  U.S. 

2  See  estimates  for  the  AFDC-FG  (female-headed)  in  Table  5  below. 
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counties  that  emphasized  human  capital  acquisition  (usually  involving  longer  periods  of  basic 
education  and  training),  Riverside  emphasized  a  tightly  focused  job  search  program  (known  as 
“Job  Club”)  and  maintained  a  consistent  message  “that  employment  is  central  and  should  be 
sought  expeditiously  and  that  opportunities  to  obtain  low-paying  jobs  should  not  be  turned 
down.”3 

The  “work-first”  approach  of  Riverside,  which  received  national  (and  international)  ac¬ 
claim  for  its  success,4  has  become  the  standard-bearer  and  model  for  welfare-to-work  programs 
not  only  for  California  but  also  for  the  rest  of  the  nation.5  The  emphasis  in  the  work-first  ap¬ 
proach  on  placing  people  into  jobs  quickly,  even  if  at  low  initial  wages,  reflects  a  view  that  the 
workplace  is  where  welfare  recipients  can  best  acquire  their  work  habits  and  skills.  However, 
there  are  several  reasons  why  the  findings  from  MDRC’s  evaluation  of  Riverside’s  GAIN  pro¬ 
gram  do  not  necessarily  imply  that  the  work-first  strategy  is  more  effective  than  the  human- 
capital  strategy  for  increasing  the  self-sufficiency  and  reducing  the  welfare  dependence  of  recipi¬ 
ents. 

The  first  issue  concerns  the  timeframe  over  which  the  effects  of  these  types  of  programs 
are  typically  measured.  As  noted  above,  the  MDRC  has  published  estimates  of  GAIN  impacts 
only  for  the  first  three  years  after  random  assignment  to  treatment.  As  a  general  matter,  extrapo¬ 
lating  from  short  run  estimates  of  the  impacts  of  social  programs  to  what  will  happen  in  the 
longer  run  can  be  misleading,  as  Couch  (1992)  and  Friedlander  and  Burtless  (1995)  have  noted. 
More  importantly  for  the  case  at  hand,  reliance  on  short-run  estimates  of  program  effects  will 

3  Hogan  (1995). 

4  For  example,  the  Riverside  GAIN  program  was  awarded  the  Harvard  Kennedy  School  of  Government’s  “Innova¬ 
tions  in  American  Government  Award”  in  1996. 

5  Based  on  these  evaluations  of  the  GAIN  programs,  the  Wilson  administration  in  California  pushed  to  have  all  of 
the  state’s  counties  adopt  the  Riverside  work-first  approach  in  its  GAIN  programs,  culminating  in  the  1995  GAIN 
reforms  (AB  1371). 
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tend  to  understate  effectiveness  of  human  capital  development  programs  relative  to  those  that 
emphasize  early  labor  force  entry  and  attachment,  simply  because  the  human-capital  develop¬ 
ment  programs  are  more  time- intensive  treatments  and  typically  take  longer  to  complete  relative 
to  work-first  programs.  Work-first  programs  typically  take  only  a  few  weeks  for  participants  to 
complete  and  such  programs  concentrate  on  getting  workers  into  jobs  as  soon  as  possible.  As 
such,  there  is  a  strong  presumption  that,  relative  to  long-term  results,  short-term  evaluations  will 
favor  work-first  programs  over  human-capital  development  ones.6  Estimates  of  program  effects 
over  a  longer  post-enrollment  period  are  needed  before  one  can  accurately  assess  relative  long- 
run  benefits  of  these  alternative  welfare-to-work  strategies. 

The  second  reason  for  caution  in  inferring  the  relative  effectiveness  of  alternative  training 
strategies  relates  to  the  design  of  the  GAIN  evaluation.  We  want  to  compare  the  outcomes  of  two 
different  treatments,  namely,  HCD  and  LFA.  To  do  so,  ideally,  the  experiment  would  have  ran¬ 
domly  assigned  individuals  in  the  same  location  to  the  two  programs  or  to  a  control  group.  In¬ 
stead,  AFDC  assistance  units  within  a  county  were  randomly  assigned  either  to  receive  the  ser¬ 
vices  of  the  county’s  particular  implementation  of  GAIN  or  to  be  denied  these  services.  As 
MDRC  made  clear  in  its  reports  on  this  evaluation,  this  experimental  design  does  not  allow  one 
to  draw  inferences  about  the  differential  impact  of  alternative  programs — e.g.,  work-first  versus 
human-capital — that  vary  between  counties  with  the  same  level  of  rigor  that  apply  to  the  within- 
county  gross  impacts  of  a  county-specific  program  relative  to  no  program.  This  is  because  pro¬ 
gram  effects  may  he  heterogeneous  across  individuals  and  programs  in  different  counties  may 
have  selected  different  mixes  of  participants.  Alternatively,  program  effects  may  vary  with  eco- 

6  A  similar  point  is  made  by  Mincer  (1974)  in  his  model  of  schooling  decisions.  Therein,  Mincer  notes  that  at  early 
ages  the  earnings  of  individuals  who  choose  additional  schooling  will  be  lower  than  those  who  choose  to  go  to  work 
at  early  ages,  simply  because  attending  school  inhibits  going  to  work,  even  if  all  alternative  activities  yield  the  same 
present  value  of  lifetime  earnings. 
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nomic  conditions  and  these  conditions  may  vary  across  the  counties.  Moreover,  even  randomiza¬ 
tion  over  the  three  types  of  treatments  considered  in  the  MDRC  GAIN  Evaluation  namely 
HCD,  LFA  and  no  services — would  only  allow  one  to  assess  the  efficacy  of  a  program  within  a 
particular  site.  The  ability  to  assess  the  policy  question  of  interest — whether  future  implementa¬ 
tions  of  welfare-to-work  programs  should  follow  a  work-first  or  human  capital  development 
strategy— will  still  depend  on  the  credibility  of  extrapolating  results  from  site  to  other. 

In  this  paper  we  address  both  of  these  concerns.  To  address  whether  estimates  of  the 
GAIN  impacts  based  on  only  three  years  of  data  are  indicative  of  the  longer  run  gains  from 
GAIN  programs,  we  present  estimates  of  the  impacts  using  nine  years  of  post-enrollment  data  on 
the  employment,  earnings  and  welfare  participation  of  the  members  of  the  experimental  and  con¬ 
trol  groups  in  four  urban  counties  used  in  the  MDRC  evaluation  of  the  California  GAIN  program 
(Alameda,  Los  Angeles,  Riverside  and  San  Diego  counties).7  Our  estimates  of  longer-run  im¬ 
pacts  exploit  the  within-county  random  assignment  design  of  the  original  MDRC  evaluation  and, 
therefore,  maintain  the  credibility  ascribed  to  such  a  design  for  the  gross  impacts  of  each  of 
GAIN  programs  implemented  in  these  counties  for  the  populations  they  served. 

To  address  the  second  issue — generalizing  the  findings  from  one  county  to  another 
county  and  comparing  the  effects  of  HCD  and  LFA  strategies— the  paper  also  provides  estimates 
of  the  differential  impacts  of  the  work-first  strategy  implemented  in  Riverside  county  relative  to 
the  more  human-capital  oriented  programs  used  in  the  other  three  counties  at  the  time  of  random 
assignment  in  the  original  MDRC  evaluation.  In  principle,  the  ideal  way  of  estimating  such  dif¬ 
ferential  impacts  within  a  county  would  be  to  randomly  assign  subjects  from  each  of  the  counties 
to  one  of  three  arms:  (a)  work-first,  (b)  human-capital  treatments,  or  (c)  a  control  group  that  re- 

7  We  omit  the  two  rural  counties  included  in  the  original  MDRC  evaluation,  (Butte  and  Tulare),  because  these  rural 
economies  are  quite  different  from  the  four  urban  counties. 
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ceives  neither  bundle  of  services.  Currently,  data  for  such  an  experiment  does  not  exist,  at  least 
not  one  with  data  on  long-term  post-enrollment  outcomes.8  Furthermore,  implementing  such  a 
design  is  difficult  in  practice,  especially  when  a  particular  treatment  entails  sending  and  main¬ 
taining  an  overarching  orientation,  or  “message.”  For  example,  it  has  been  strongly  suggested 
that  an  essential  feature  of  the  Riverside  GAIN  program  was  the  pervasiveness  of  their  “mes¬ 
sage”  that  all  program  participants,  even  those  with  skill  deficits,  should  strive  to  get  a  job 
quickly,  regardless  of  its  compensation  (e.g.  Corbett,  1994/95).  Evaluation  designs  with  multiple 
treatments,  in  which  one  or  more  treatments  attempt  to  maintain  such  programmatic  “messages” 
or  “orientations”  would  appear  to  be  nearly  impossible  to  implement  within  a  single  bureaucracy 
or  agency  or  on  a  sufficient  scale. 

To  estimate  the  differential  impacts  of  alternative  programs,  we  instead  make  use  of  sta¬ 
tistical  matching  and  regression  adjustments  based  on  the  personal  characteristics,  past  welfare, 
and  earnings  histories  of  welfare  recipients  in  the  four  counties  in  an  attempt  to  adjust  for  across- 
county  differences  in  GAIN  participants.  While  one  cannot  claim,  a  priori,  that  such  adjustments 
eliminate  these  across-county  differences  in  the  participant  populations,  we  exploit  the  availabil¬ 
ity  of  data  on  control  groups  denied  all  GAIN  services  in  each  of  the  counties  to  assess  the  qual¬ 
ity  and  credibility  of  these  adjustments.  As  a  result  of  the  random  assignment  of  welfare  recipi¬ 
ents  eligible  for  GAIN  services  in  each  county,  the  control  groups  reflect,  on  average,  each 
county’s  participant  pool  in  the  absence  of  receiving  the  GAIN  treatments  available  in  each  of 
the  counties.  Thus,  we  perform  statistical  tests  of  whether  our  adjustments  eliminate  across- 
county  differences,  on  average,  in  county  post-enrollment  outcomes  for  controls.9  If  our  regres- 

8  We  note  that  MDRC  is  conducting  such  an  evaluation  in  three  counties  in  the  U.S.  in  their  National  Evaluation  of 
Welfare-to-Work  Strategies.  To  date,  results  for  two  years  of  post-enrollment  outcomes  have  been  released.  See 
Hamilton,  et  al.  (1997). 

9  Dehejia  (2000)  discusses  the  related  question  whether  data  from  different  sites  can  be  pooled  conditional  on  co- 
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sion  adjustments  are  sufficient,  there  should  be  no  remaining  differences  between  the  outcomes 
of  control  group  members,  who  received  no  GAIN  services,  across  the  counties.  We  note  that  the 
strategy  of  adjustment,  including  matching,  and  validating  these  methods  by  using  data  for  ex¬ 
perimentally-generated  control  groups  is  similar  to  the  strategies  used  by  Lalonde  (1986), 
Heckman  and  Hotz  (1989),  Heckman,  Ichimura,  and  Todd  (1997,  1998a,  1998b),  Dehejia  and 
Wahba  (1999),  and  Hotz,  Imbens  and  Mortimer  (1999). 

We  then  use  the  estimated  models  to  address  the  question  of  differential  impacts  of  alter¬ 
native  programs.  The  models  estimate  the  effect  of  treatment  for  a  vector  of  observed  character¬ 
istics.  We  use  the  predictions  of  the  model  to  estimate  the  effect  of  applying  the  treatment  in 
Riverside  County  to  the  treated  population  in  the  other  three  counties.  Given  the  9-year  post-en¬ 
rollment  data  on  outcomes  that  we  have,  we  can  assess  both  the  short-  and  longer-run  differential 
impacts  of  these  two  types  of  treatment  strategies  for  the  various  county-specific  treatment  popu¬ 
lations.  These  predicted  effects  for  two  programs,  for  a  given  population ,  can  then  be  compared 
to  assess  the  differential  effects  of  types  of  services  (treatments)  used  in  the  respective  programs. 
We  show  that  these  matching  estimates  differ — sometimes  substantially — from  the  simple  com¬ 
parison  of  the  net  treatment  effects  in  each  county,  suggesting  that  effect  heterogeneity  is  sub¬ 
stantively  important. 

The  remainder  of  the  paper  is  organized  as  follows.  In  Section  2,  we  provide  a  brief  de¬ 
scription  of  California’s  GAIN  program  and  the  original  MDRC  evaluation.  We  begin  by  show¬ 
ing  that  the  populations  selected  (i.e.,  who  were  subject  to  randomization)  were  very  different 
across  the  counties;  Riverside,  and  to  a  lesser  extent,  San  Diego,  chose  to  enroll  in  their  GAIN 
programs  nearly  all  cases,  the  other  counties  choosing  to  enroll  in  their  GAIN  program  only  long 
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term  welfare  recipients  who  are  potentially  the  most  difficult  to  serve.  Thus,  if  there  is  treatment 
heterogeneity,  it  is  likely  to  affect  our  comparison  of  program  effects  across  counties.  In  Section 
3,  we  present  within-county  experimental  estimates  of  the  gross  impacts  of  the  county-specific 
GAIN  programs  on  the  employment,  earnings,  and  welfare  participation  during  the  9-year  post¬ 
enrollment  period.  In  Section  4,  we  provide  a  more  detailed  discussion  of  our  strategy  for  esti¬ 
mating  the  differential  impacts  of  the  Riverside  GAIN  program  relative  to  the  GAIN  programs 
used  in  Alameda,  Los  Angeles,  San  Diego  counties  and  present  estimates  for  the  same  outcomes 
and  follow-up  period  as  analyzed  in  Section  3.  Finally,  we  offer  some  conclusions  about  the  im¬ 
plications  of  our  findings  in  Section  5. 

2.  The  GAIN  Program,  the  MDRC  Evaluation  and  GAIN  Evaluation  Counties 

In  this  section  we  provide  a  brief  description  of  the  structure  of  the  GAIN  program  and 
how  it  was  implemented  in  the  four  urban  counties  we  consider  in  this  paper.  We  also  describe 
the  structure  of  the  MDRC  GAIN  Evaluation. 

The  GAIN  program  began  in  California  in  1986  and,  in  1989,  became  the  state’s  official 
Job  Opportunities  and  Basic  Skills  Training  (JOBS)  Program,  authorized  by  the  Family  Support 
Act,  the  nation’s  attempt  to  reform  the  welfare  system  prior  to  PRWORA.  The  GAIN  program 
represented  a  compromise  between  two  groups  in  the  State’s  legislature  with  different  visions  of 
how  to  reform  the  welfare  system.  One  group  favored  the  “work-first”  approach,  i.e.,  use  of  a 
relatively  short-term  program  of  mandatory  job  search,  followed  by  unpaid  work  experience  for 
participants  who  did  not  find  jobs.  The  other  group  favored  the  “human  capital”  approach,  i.e.,  a 
program  providing  a  broader  range  of  services  designed  to  develop  the  skills  of  welfare  recipi¬ 
ents. 

In  crafting  the  GAIN  legislation,  these  two  groups  compromised  on  a  program  that  con- 
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tained  work-first  as  well  as  basic  skills  and  education  components  in  what  became  known  as  the 
GAIN  Program  Model.10  The  GAIN  model  consisted  of  the  following  sequence  of  steps.  At  the 
time  of  initial  (or  continuing)  determination  of  eligibility  for  welfare,  county  staff  also  deter¬ 
mined  whether  the  head  of  a  welfare  household  was  subject  to  GAIN11  and,  if  so,  registered  her 
(usually,  although  sometimes  him)  for  GAIN.  (Staff  also  offered  to  register  adults  on  welfare 
who  were  exempted  but  wished  to  volunteer  for  the  program.)  A  county’s  GAIN  registrant  was 
required  to  attend  an  orientation  meeting  to  leam  about  the  county’s  particular  GAIN  program 
and  their  obligations  under  this  program.  Each  registrant  was  administered  a  screening  test  to 
measure  a  registrant’s  basic  reading  and  math  skills.  (The  same  test  was  used  in  each  of  the 
counties.)  Based  on  their  score  on  this  test  and  whether  she  had  a  high  school  diploma  or  a  GED, 
she  was  sent  to  one  of  two  service  tracks.  A  registrants  with  a  low  test  score  and  who  did  not 
have  a  high  school  diploma  or  GED  was  deemed  “in  need  of  basic  education”  and  was  to  be 
routed  through  a  sequence  of  services  that  included  access  to  Adult  Basic  Education  (ABE)  or 
English  as  a  Second  Language  (ESL)  programs.  Those  not  judged  to  be  in  need  of  basic  educa¬ 
tion  were  to  bypass  these  basic  education  services.  A  registrant,  in  either  group,  was  then  to  be 
channeled  into  job  search  activities  in  an  attempt  to  get  her  employed.  If  the  registrant  did  not 
find  a  job,  they  were  to  be  provided  access  to  vocational,  on-the-job  training  and  work  experi¬ 
ence  activities,  in  an  attempt  to  enhance  a  registrant’s  human  capital  and,  thus,  to  improve  their 
chances  of  securing  a  job. 

While  the  GAIN  legislation  set  out  a  clear  set  of  goals  for  the  program  and  the  above 
model  for  the  delivery  of  services,  it  also  gave  California’s  58  counties  substantial  discretion  and 

10  See  Riccio  and  Friedlander  (1992)  for  a  more  complete  description  of  this  model. 

11  Heads  of  households  on  welfare  were  mandated  to  register  for  GAIN,  except  for  female  heads  with  children  under 
the  age  of  6.  See  Riccio,  et  al.  (1989)  for  a  more  complete  description  of  the  criteria  for  mandated  participation 
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flexibility  in  designing  their  programs.  In  particular,  counties  had  discretion  over  the  types  of 
welfare  recipients  they  registered  for  their  GAIN  programs  and  the  relative  weight  they  placed 
on  quick  labor  market  entry  versus  skill  development.12  County  GAIN  programs  differed  along 
both  of  these  dimensions.  With  respect  to  the  types  of  welfare  recipients  registered  for  GAIN, 
some  counties,  like  Riverside  and  San  Diego,  operated  “universal”  programs  in  that  all  welfare 
applicants  and  recipients  were  registered  for  GAIN,  while  others,  like  Alameda  and  Los  Angeles, 
registered  mostly  long-term  welfare  recipients,  who  presumably  were  more  difficult  to  serve. 

MDRC  conducted  a  randomized  evaluation  of  the  impacts  and  cost-effectiveness  of  the 
GAIN  program  in  six  research  counties  (Alameda,  Butte,  Los  Angeles,  Riverside,  San  Diego, 
and  Tulare).  From  the  latter  part  of  1988  to  the  middle  of  1990,  each  county  chose  whom  to  reg¬ 
ister  for  GAIN.  MDRC  then  randomly  assigned  some  of  these  registrants  to  an  experimental 
group,  which  was  eligible  to  receive  GAIN  services  and  subject  to  its  participation  mandates,  and 
the  remainder  of  these  registrants  to  a  control  group,  whose  members  were  not  eligible  for  GAIN 
services  or  mandates  but  could  seek  (on  their  own  initiative)  alternative  services  in  their  commu¬ 
nities.  Note  that  because  the  counties  followed  different  practices  with  respect  to  choosing  regis¬ 
trants,  both  the  experimental  and  control  populations  will  vary  across  the  research  counties.  The 
controls  were  embargoed  from  any  GAIN  services  from  the  date  of  their  random  assignment  un¬ 
til  June  30,  1993. 13  MDRC  collected  data  on  both  experimental  and  control  group  members  in 
each  of  the  research  counties,  including  some  background  and  demographic  characteristics  and 
on  a  set  of  outcomes  after  random  assignment.  (Most  of  these  data  were  obtained  from  state  and 
county  administrative  data  systems.)  They  also  monitored  the  operations  of  the  programs  in  each 

12  See  Riccio  and  Friedlander  (1992),  chapter  1. 

13  For  a  period  of  two  years  following  the  lifting  of  this  embargo,  control  group  members  were  not  required  to  par¬ 
ticipate  in  GAIN  but  could  if  they  asked  to  enroll. 
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of  the  six  research  counties.  MDRC  issued  a  series  of  reports  on  program  operation14  and  on  the 
impacts  of  GAIN  programs  in  these  counties  over  the  three-  to  five-year  post  random  assignment 

period.15 

Descriptive  statistics  and  sample  sizes  for  the  participants  in  the  MDRC  evaluation  in  the 
four  analysis  counties  (Alameda,  Los  Angeles,  Riverside  and  San  Diego)  are  provided  in  Table 
1.  Separate  statistics  are  provided  for  GAIN  registrants  who  were  members  of  female-headed 
households  on  AFDC  at  the  time  of  random  assignment  (AFDC-FG  cases)  and  two-parent 
households  on  AFDC  (AFDC-U  cases).  The  statistics  in  Table  1  will  be  discussed  below.  We 
note  that  our  samples  for  three  of  the  four  counties  (all  but  Los  Angeles)  are  slightly  smaller  than 
the  original  samples  used  by  MDRC  due  to  our  inability  to  find  records  for  some  sample  mem¬ 
bers  in  California’s  Unemployment  Insurance  Base  Wage  system16  or  because  we  were  missing 
information  on  the  educational  attainment  of  the  sample  member.  The  number  of  cases  lost  in 
these  three  counties  is  very  small,  never  larger  than  1.8%  of  the  total  sample,  and  does  not  ap¬ 
pear  to  differ  by  experimental  status.  Finally,  note  that  in  most  of  the  counties  and  especially  for 
AFDC-FG  cases,  a  much  larger  fraction  of  cases  were  assigned  to  the  experimental  group  than  to 
the  control  group. 

As  noted  above,  there  are  differences  in  what  types  of  AFDC  cases  were  registered  for 
GAIN.  For  the  four  counties  we  analyze,  the  programs  in  Riverside  and  San  Diego  counties 
sought  to  register  all  welfare  cases  in  GAIN  while  the  programs  in  Alameda  and  Los  Angeles 


14  See  Wallace  and  Long  (1987),  Riccio,  et  al.  (1989)  and  Riccio  and  Friedlander  (1992). 

15  See  Riccio  and  Friedlander  (1992),  Riccio,  et  al.  (1994)  and  Freedman,  et  al.  (1996). 

16  The  California  Economic  Development  Department  (EDD)  administers  the  State  s  UI  system. 

17  The  losses  from  the  original  MDRC  samples  were  as  follows:  Alameda:  0.6%  for  AFDC-FGs  and  6.0%  for 
AFDC-Us;  Los  Angeles:  0.02%  for  AFDC-FGs  and  0.0%  for  AFDC-Us;  Riverside:  1.1%  for  AFDC-FGs  and  1.8% 
for  AFDC-Us;  San  Diego:  1.1%  for  AFDC-FGs  and  1.0%  for  AFDC-Us. 
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counties  focused  on  long-term  welfare  recipients.18  The  consequences  of  these  differences  in  se¬ 
lection  criteria  can  be  seen  in  Table  1.  Panel  A  of  Table  1  displays  the  characteristics  of  the 
AFDC-FG  cases,  i.e.,  households  with  a  single  adult  (usually  a  female)  on  welfare,  while  Panel 
B  provides  comparable  information  for  AFDC-U  cases,  i.e.,  households  with  a  married  couple. 
In  Alameda  and  Los  Angeles,  over  95%  of  the  cases  had  been  on  welfare  a  year  prior  to  random 
assignment;  in  San  Diego  and  Riverside,  fewer  (for  some  cells  much  fewer)  than  65  percent  had 
been. 

These  differences  in  selection  criteria  also  contributed  to  substantial  differences  in  the 
employment  histories  and  individual  characteristics  of  the  registrant  populations  across  these 
four  counties.  As  shown  in  the  two  Panels  of  Table  1,  the  registrants  in  Alameda  and  Los  Ange¬ 
les  counties  had,  on  average,  much  lower  levels  of  earnings  prior  to  random  assignment  relative 
to  those  in  Riverside  and  San  Diego.  These  differences  in  past  income  across  counties  are  most 
dramatic  for  AFDC-U  cases,  where  the  average  past  earnings  of  GAIN  registrants  in  Riverside 
and  San  Diego  counties  were  between  5  and  7  times  greater  than  those  for  Alameda  and  Los  An¬ 
geles  registrants.  Furthermore,  the  registrants  in  Alameda  and  Los  Angeles  were,  on  average, 
older,  had  lower  levels  of  educational  attainment,  and  were  more  likely  to  be  assessed  as  “in 
need  of  basic  education”  when  they  entered  the  GAIN  program  than  the  average  registrants  in 
Riverside  and  San  Diego.  The  differences  in  characteristics  of  GAIN  registrants  displayed  in  the 
two  panels  of  Table  1  also  suggest  the  possibility  of  differences  in  the  overall  low-income  and 
welfare-prone  populations  that  reside  in  each  of  these  counties. 

The  across-county  differences  in  earnings,  labor  force  and  welfare  participation  rates  for 
GAIN  registrants  also  may  have  resulted  from  differences  in  the  labor  market  conditions  pre- 

15  For  example,  Alameda  County,  which  began  its  GAIN  program  in  the  third  quarter  of  1989,  began  by  registering 
cases  that  had  been  receiving  AFDC  since  1989,  subsequently  registering  more  recent  recipients.  The  GAIN  pro- 
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vailing  at  the  time  of  random  assignment.  In  Table  2,  we  present  data  on  the  labor  market  condi¬ 
tions  in  the  four  counties  for  the  years  during  which  GAIN  registrants  were  enrolled  into  the 
MDRC  GAIN  Evaluation.  While  it  is  unclear  the  extent  to  which  these  differences  in  labor  mar¬ 
ket  conditions  can  account  for  the  across-county  differences  in  earnings,  employment  and  wel¬ 
fare  participation  of  GAIN  registrants  noted  in  Table  1,  the  diversity  in  the  labor  markets  of  these 
counties  is  quite  apparent.  Table  2  reveals  noticeable  differences  in  the  structure  and  state  of  the 
labor  markets  in  these  counties  around  the  enrollment  period.  For  example,  both  Alameda  and 
Los  Angeles  Counties  had  higher  shares  of  employment  in  the  manufacturing  sector  than  did  ei¬ 
ther  Riverside  or  San  Diego  Counties.  Around  the  time  of  the  enrollment  into  the  GAIN  Evalua¬ 
tion,  employment  in  Riverside  County  was  growing  at  a  much  higher  rate  (7.4  to  8.5%)  than  was 
the  case  in  any  of  the  other  three  counties. 

There  also  were  marked  differences  in  the  policies  and  practices  that  were  followed  in 
these  counties.  Senior  leadership  in  the  four  counties  had  very  different  views  about  what  ser¬ 
vices  to  provide  to  welfare  recipients  and  the  likely  cost-effectiveness  of  those  services.  The 
GAIN  legislation  provided  the  counties  with  enough  discretion  to  allow  them  to  allocate  re¬ 
sources  consistent  with  those  views  thus  leading  to  very  different  programs.  In  particular,  coun¬ 
ties  differed  in  the  emphasis  placed  on  work-first  versus  human  capital  and  skill  development  in 
their  GAIN  programs.  Riverside’s  program  stood  apart  from  other  counties  in  degree  to  which 
staff  emphasized  moving  registrants  into  the  labor  market  quickly.  This  difference  in  Riverside’s 
work-first  orientation  is  reflected  in  the  distribution  of  program  activities  over  the  first  three 
years  of  GAIN’S  operation  (see  Table  3).  (The  shaded  quarters  in  this — and  the  next— table  show 
the  quarters  in  which  the  random  assignment  of  registrants  into  the  MDRC  experimental  evalua- 

gram  in  Los  Angeles  County  initially  only  registered  those  cases  that  had  been  on  welfare  for  3  consecutive  years. 
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tion  were  conducted  for  each  of  the  four  counties.) 

The  activities  in  the  table  are  organized  into  groups,  one  representing  job  search-related 
activities,  another  consisting  of  basic  skills  and  educational  activities,  and  a  third  including  ac¬ 
tivities  which  provided  registrants  with  direct  work  experience.  Clearly,  Riverside  disproportion¬ 
ately  channeled  its  registrants  into  job  search  activities  relative  to  basic  skills  activities.  (None  of 
the  county  programs  made  extensive  use  of  work-experience  activities  in  the  early  stages  of  their 
operation.)  Riverside’s  emphasis  on  job  search  activities  stands  in  contrast  to  the  other  three 
counties,  especially  Alameda  and  Los  Angeles,  where  registrants  were  much  more  likely  to  be  in 
basic  skills  activities  in  any  given  month. 

The  data  in  Table  3  are  consistent  with  other  indicators  of  Riverside’s  emphasis  on  get¬ 
ting  GAIN  registrants  quickly  into  jobs.  For  example.  Riverside  staff  required  that  their  regis¬ 
trants  that  were  enrolled  in  basic  skills  programs  continue  to  participate  in  Job  Club  and  other  job 
search  activities.  In  a  survey  of  program  staff  conducted  by  MDRC  at  the  time  of  its  evaluation, 
95%  of  case  managers  in  Riverside  rated  getting  registrants  into  jobs  quickly  as  their  highest 

19 

goal  while  fewer  than  20%  of  managers  in  the  other  research  counties  gave  a  similar  response. 

In  the  same  survey,  69%  of  Riverside  case  managers  indicated  that  they  would  advise  a  welfare 
mother  offered  a  low-paying  job  to  take  it  rather  than  wait  for  a  better  opportunity,  while  only 
23%  of  their  counterparts  in  Alameda  county  indicated  they  would  give  this  advice. 

Overall,  MDRC  concluded,  “What  is  perhaps  most  distinctive  about  Riverside’s  program, 
though,  is  not  that  its  registrants  participated  somewhat  less  in  education  and  training,  but  that 
the  staffs  emphasis  on  jobs  pervaded  their  interactions  with  registrants  throughout  the  program” 
(Riccio  and  Friedlander,  1992,  p.  58).  Riverside  County’s  GAIN  staff  were  instructed  to  com- 

19  See  Table  3.1  in  Riccio  and  Friedlander  (1992)  for  the  results  of  this  survey. 
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municate  a  strong  “message”  to  all  registrants,  including  those  in  education  and  training  activi¬ 
ties,  that  gaining  employment  was  central,  that  it  should  be  sought  expeditiously,  and  that  jobs 
should  not  be  turned  down  even  if  they  were  low-paying.  In  contrast,  program  staff  in  the  other 
research  counties  placed  less  emphasis  on  getting  registrants  into  a  job  quickly.  For  example, 
Alameda’s  GAIN  managers  and  staff  “believed  strongly  in  ‘human  capital’  development  and, 
within  the  overall  constraints  imposed  by  the  GAIN  model’s  service  sequences,  its  staff  encour¬ 
aged  registrants  to  be  selective  about  the  jobs  they  accepted  and  to  take  advantages  of  GAIN’S 

s20 

education  and  training  to  prepare  for  higher-paying  jobs.” 

A  final  indicator  of  the  differences  in  the  way  Riverside’s  GAIN  program  operated,  rela¬ 
tive  to  the  programs  in  the  other  counties,  can  be  seen  in  Table  4.  This  table  displays  the  average 
monthly  GAIN  enrollments,  by  county,  as  a  percentage  of  each  county’s  AFDC  caseload.  Com¬ 
pared  to  the  programs  in  Alameda  and  Los  Angeles,  Riverside  consistently  provided  GAIN  ser¬ 
vices  to  more  of  its  caseload.  This  pattern  for  these  three  counties  is  consistent  with  the  fact  that 
the  latter  two  counties  enrolled  more  of  their  registrants  in  basic  skills  and  education  programs 
compared  to  those  focused  on  job  search.  The  former  programs  are,  on  average,  much  more  ex¬ 
pensive,  on  a  per  case  basis,  compared  to  the  latter  activities.  We  note  from  Table  4  that  San 
Diego  actually  enrolled  an  even  higher  proportion  of  its  AFDC  caseload  in  GAIN  activities  than 
any  other  county,  including  Riverside.  This  reflected  the  fact  that  San  Diego  officially  “enrolled” 
a  large  number  (and  percentage)  of  its  AFDC  participants  in  GAIN  even  though  most  of  these 
registrants  did  not  participate  in  any  activities.  Rather,  they  remained  in  a  queue,  waiting  until 
slots  in  services,  provided  by  an  outside  contractor,  became  available. 

All  of  the  evidence  provided  above  clearly  suggests  that  the  prevailing  “treatment”  in 


20  See  Riccio,  et  al.  (1994),  p.  xxv. 
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Riverside  County’s  GAIN  program — both  in  terms  of  way  it  distributed  its  registrants  across  ac¬ 
tivities  and  in  the  pervasive  message  it  provided  to  them — was  one  that  had  a  “work-first”  ori¬ 
entation,  while  the  other  county  programs  we  consider  in  this  paper,  especially  the  Alameda  and 
Los  Angeles  programs,  disproportionately  provided  their  registrants  with  a  human  capital,  skill 
development  oriented  treatment. 

3.  Estimating  County-Specific  Effects  of  GAIN  Programs 

In  this  section  we  discuss  the  experimental  estimates  of  the  impacts  of  GAIN  services  and 
mandates  to  which  GAIN  registrants  were  subject  during  the  early  1990s  on  their  employment, 
earnings  and  welfare  participation  for  up  to  nine  years  after  random  assignment.  While  well 
known,  we  briefly  characterize  the  properties  of  experimental  estimators  of  such  impacts  in  an¬ 
ticipation  of  our  discussion  of  the  estimation  of  the  differential  effects  of  work-first  versus  hu¬ 
man  capital  development  treatments  in  Section  4.  We  then  present  the  estimates  of  the  short-  and 
long-run  impacts  for  each  of  the  four  counties  using  the  MDRC  GAIN  Evaluation  samples. 

3.1  Identifying  Within-County  Treatment  Effects 

To  help  fix  ideas,  we  define  the  following  notation.  Let  D  denote  an  indicator  of  the 

county  (and  its  GAIN  program),  where  d  e  {A,L,R,S}  for  the  four  counties  in  our  study.  Random 
samples,  of  size,  Nd,  are  drawn  from  the  GAIN  program  registrants  in  each  county  d  and  let  /  de¬ 
note  a  household  in  these  samples.  Let  t  denote  the  period  (year)  after  a  household  has  been  ran¬ 
domly  assigned.  Let  T  denote  the  treatment  indicator  for  the  treatments  under  the  MDRC  GAIN 
evaluation,  where  T  e  {0,  w,  h},  where  0  denotes  no  GAIN  services,  w  denotes  the  work-first 
oriented  treatment  and  h  denotes  the  human-capital  development  treatment.  Let  Yit(k)  denote  the 
potential  outcomes  for  household  i  in  t  periods  after  their  (random)  assignment  to  treatment  k,  so 
that  7„(0)  is  the  potential  outcome  associated  for  the  no-treatment  case,  Yit(w)  is  the  potential 
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outcome  associated  with  the  work-first  treatment  and  YuQi )  is  the  potential  outcome  for  the  hu¬ 
man-capital  development  treatment  as  of  period  t.  Finally,  let  Xj  denote  a  vector  of  background 
characteristics  and  pre-treatment  variables  for  household  i. 

Access  to  experimental  data  allows  us  to  estimate  the  average  gross  treatment  effect 
(AGTE)  of  treatment  k  on  those  treated,  which  is  defined  to  be 

«,(*) -  E(Y„(k)-Y,XD)\T1  - *)  =  £(A„(*)|7;  =  *).  0) 

where  A„(Jfc)  (=  7„(0)  -  Yfk))  is  household  f  s  gain  as  of  period  t  from  treatment  k  relative  to  re¬ 
ceiving  no  GAIN  services.  The  conditional  (on  X)  version  of  this  treatment  effect  is  given  by, 

a,  (k  | *)  s  E(A„  (k))  if  =  k ,  X, .  =  x) ,  (2) 

for  all  k  €  {w,h}.  As  noted  above,  the  design  of  the  MDRC  GAIN  Evaluation  was  to  randomly 
assign  GAIN  registrants  in  each  County  d  to  either  the  prevailing  treatment  in  that  county,  T,  =  kd 
or  to  a  control  group  that  received  no  services,  Tt  =  0.  This  design  implies  that 

Y„(T)  =  d ,  (C-l) 

where  z±y  denotes  that z  is  (statistically)  independent  ofy,  where  in  (C-l)  the  independence  is 
conditional  on  the  county  of  residence  (D,  =  d).  Note  that  (C-l)  implies  that 

E(Y„(0)\f  =  k,Di=d)  =  E(Yll(0)\Ti=0,D,=d),  (C-l') 

i.e.,  the  mean  value  of  7(0)  for  those  who  receive  treatment  k  in  County  d  is  equal  to  the  mean 
value  of  the  outcomes  as  of  period  t  for  control  group  members  in  that  same  county.  Note  that 
the  latter  is  typically  unobserved  by  the  econometrician  while  the  latter  is  observable  with  data 
from  an  experiment  with  random  assignment  of  treatments.  MDRC’s  evaluation  design  guaran¬ 
tees  that  treatment  received  by  individual  i  is  independent  of  that  unit’s  potential  outcomes,  for 
all  i  within  a  particular  County  d.  Given  (C-l)  (or  (C-l')),  the  difference  between  average  out- 
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comes  for  treatment  group  members  (receiving  kj)  in  County  d  and  average  outcomes  for  the 
control  group,  also  in  County  d  and  receiving  treatment  0,  identifies  the  AGTE  for  that  county’s 
registrant  population.  That  is 

aJkJ)  =  E(r„(kJ)\T,=k,D,=d)-E(,r„(0)\T,  =0  ,D,=d),  (3) 

where  ald(kd)  denotes  the  AGTE  for  County  d,  for  all  d  e  {A,L,R,S} . 

As  discussed  in  Section  2,  GAIN  registrants  were  randomly  assigned  to  either  receive 
some  subsequently-determined  set  of  GAIN  services  or  no  services  at  all,  where  the  services  the 
“experimental”  subjects  received  were  based  on  the  GAIN  model  and  the  types  of  services  a  par¬ 
ticular  county  emphasized.  Recall  that  Riverside’s  GAIN  program  emphasized  work-first  treat¬ 
ments  ( T  =  w),  the  GAIN  programs  in  Alameda  and  Los  Angeles  Counties  emphasized  human- 
capital  development  treatments  (T  =  h ),  while  San  Diego’s  GAIN  program  was  a  mixture  of 
these  two  types  of  programs.  Moreover,  we  do  not  have  data  on  precisely  what  set  of  services 
each  GAIN  experimental  subject  actually  received.  Therefore,  in  the  next  section  we  present  es¬ 
timates  of  the  gross  effects  of  receiving  any  set  of  services  versus  no  services  for  each  of  the 
counties  instead  of  estimates  of  specific  sets  of  treatments.  In  this  sense,  we  are  approximating 
the  AGTE  for  the  “prevailing”  treatment  that  was  used  in  a  particular  county  in  place  of  a  more- 
specific  set  (or  sequence)  of  services  actually  received.  That  is,  we  use  differences  in  the  sample 
means  of  outcomes  between  experimentals  (  Yt{kd ))  and  controls  (J^(0))  to  estimate  (3)  for  each 
county. 

3.2  Long-Run  Estimates  of  County-Specific  GAIN  Impacts 

Estimates  for  AFDC-FG  cases  are  presented  in  Table  5  and  the  corresponding  estimates 
for  AFDC-U  cases  are  presented  in  Table  6.  We  provide  estimates  for  six  different  outcomes:  (1) 
ever  employed  during  year;  (2)  number  of  quarters  worked  per  year;  (3)  annual  labor  market 
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earnings;  (4)  whether  a  GAIN  registrant’s  earnings  exceeded  the  income  of  a  full-time  worker 
earning  the  minimum  wage;  (5)  whether  the  registrant  received  AFDC/TANF  benefits  during  the 
year;  and  (6)  the  number  of  quarters  in  the  calendar  year  that  she  received  AFDC/TANF  bene¬ 
fits.21,22  For  each  of  these  outcomes  we  have  nine  years  of  post-random  assignment  data.  We  av¬ 
erage  them  over  three-year  periods  to  reduce  the  number  of  entries  in  tables  that  follow.  MDRC 
has  published  estimates  of  similar  outcomes  for  3  years  post  random  assignment  and  5-year  post¬ 
random  assignment  results  were  released  in  a  working  paper.  3  (The  estimates  presented  in  Ta¬ 
bles  4  and  5  for  the  first  five  post  random  assignment  years  do  not  correspond  exactly  to  the  es¬ 
timates  produced  by  MDRC,  due  to  relatively  minor  differences  in  samples  used  (see  discussion 
above)  and,  more  importantly,  the  use  of  a  different  “dating”  convention  in  forming  post-random 

.  24\ 

assignment  years.  ) 

Our  access  to  data  on  outcomes  for  six  additional  years  after  random  assignment  allows 
us  to  assess  the  longer-term  consequences  of  being  exposed  to  the  GAIN  programs  in  the  four 


21  The  employment  and  earnings  outcomes  were  constructed  with  data  from  the  State’s  UI  Base  Wage  files  provided 
by  the  California  Employment  Development  Department  (EDD).  These  data  contain  quarterly  reports  from  employ¬ 
ers  on  whether  individuals  were  employed  in  a  Ul-covered  job  and  their  wage  earnings  for  that  job.  These  quarterly 
data  were  organized  into  four-quarter  “years”  from  the  quarter  of  enrollment  in  the  MDRC  GAIN  evaluation.  The 
“Ever  Employed  in  Year”  outcome  was  defined  to  be  =  1  if  the  individual  had  positive  earnings  in  at  least  one  quar¬ 
ter  during  that  year  and  =  0  otherwise.  The  “Annual  Earnings”  outcome  was  the  sum  of  the  four-quarter  Ul-covered 
earnings  recorded  for  an  individual  in  the  Base  Wage  file.  All  income  variables  were  converted  to  1999  dollars  using 
cost-of-living  deflators.  Finally,  the  indicator  variable  for  whether  an  individual’s  Ul-covered  earnings  exceeded  that 
the  earnings  from  working  full-time  (2,000  hours  per  year)  at  the  prevailing  Federal  minimum  wage  rate  ($5.15  per 
hour). 

22  The  AFDC/TANF  variables  were  constructed  using  data  from  the  California  statewide  Medi-Cal  Eligibility  Data 
System  (MEDS)  files,  which  contain  monthly  information  on  whether  an  individual  received  AFDC  (before  1998) 
or  TANF  (starting  in  1998)  benefits  in  California  during  a  month.  These  monthly  data  were  organized  into  3-month 
“quarters”  from  the  quarter  of  enrollment  in  the  MDRC  GAIN  evaluation  and  then  organized  into  “years”  since  en¬ 
rollment,  as  was  done  with  the  employment  and  earnings  data.  The  “Ever  Received  AFDC/TANF  Benefits  in  Year 
variable  was  defined  to  be  =  1  if  the  individual  received  AFDC  or  TANF  benefits  in  at  least  one  month  during  that 
year  and  =  0  otherwise. 

23  See  Riccio,  et  al.  (1994)  for  3-year  impact  estimates  and  Freedman,  et  al.  (1996)  for  estimates  based  on  five  years 
of  follow-up  data. 

24  In  their  analysis,  MDRC  defined  the  first  year  of  post-random  assignment  to  be  quarters  2  through  5,  year  two  as 
quarters  6  through  9,  etc.  In  our  analysis,  we  define  year  one  as  quarters  1  through  4,  year  two  as  quarters  5  through 
8,  etc.  This  difference  in  definitions  results  in  relatively  minor  differences  between  our  years  1  through  5  estimates 
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counties  we  consider.  As  noted  in  the  Introduction,  analyzing  the  longer-term  consequences  of 
GAIN  is  important  for  several  reasons.  By  analyzing  the  effects  of  these  county-level  programs 
over  a  longer  follow-up  period,  one  is  better  able  to  assess  the  duration  or  permanence  of  the  im¬ 
pacts  found  in  the  previous  3-year  and  5-year  analyses.  Furthermore,  analyzing  impacts  over  a 
nine-year  period  allows  us  to  better  assess  the  extent  to  which  there  are  differences  in  the  tempo¬ 
ral  pattern  of  the  returns  to  the  quick-job-entry  versus  skill  development  training  strategies  used 
by  the  different  counties  in  their  GAIN  programs.  This  issue  is  potentially  important,  as  noted 
above,  because  the  longer  training  periods  of  the  HCD  approach  imply  that  effects  are  likely  to 
be  negative  in  the  short-term  (during  the  training  period  and  shortly  thereafter).  On  the  other 
hand,  the  skills  developed  might  have  longer  lasting  effects  than  the  primarily  motivational  LFA 
approach.  (We  note  that  the  ranking  of  the  strategies  also  depends  upon  the  how  one  “discounts” 
the  future  benefits  as  well  as  on  the  relative  costs  of  each.) 

We  first  consider  the  long-term  impacts  of  the  four  GAIN  programs  for  AFDC-FG  cases 
presented  in  Table  5.  Consider  first  the  impacts  on  employment.  Regardless  of  whether  one  uses 
annual  employment  rates  or  the  number  of  quarters  employed  in  a  year,  one  finds  that  the  im¬ 
pacts  of  Riverside’s  program  are  consistently  larger,  and  statistically  significant,  relative  to  the 
effects  for  the  other  three  counties  over  the  first  three-year  period  after  random  assignment.  Over 
the  first  three  years,  the  GAIN  registrants  in  Riverside  had  annual  employment  rates  that  were, 
on  average,  13.6  percentage  points  (39%)  higher  than  members  of  the  control  group  and  worked 
0.43  more  quarters  per  year  (48%)  higher  than  did  control  group  members.  The  employment  im¬ 
pacts  of  the  GAIN  programs  in  the  GAIN  programs  of  the  other  three  counties  are  considerably 
lower  than  those  for  Riverside  and  often  are  not  statistically  significant.  This  relative  success  of 


relative  to  those  produced  by  MDRC. 
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the  Riverside  program  in  improving  the  employment  outcomes  of  GAIN  registrants  illustrates 
why  this  program,  and  its  work-first  orientation,  has  been  heralded  nationally  as  a  model  wel- 
fare-to-work  program. 

In  the  longer  run,  however,  the  employment  impacts  of  the  Riverside  GAIN  program  di¬ 
minish  in  magnitude  and  statistical  significance.  In  years  4  through  6  after  random  assignment, 
Riverside’s  GAIN  registrants  experience  a  6.9  percentage  point  annual  average  gain  in  annual 
rates  of  employment  (down  from  13.6  percentage  points)  and  0.25  quarters  worked  (down  from 
0.43  quarters)  over  their  control  group  counterparts.  For  years  7  through  9,  the  Riverside  GAIN 
registrants  have  an  average  annual  gain  of  only  1.5  percentage  points  in  annual  rates  of  employ¬ 
ment  and  0.08  quarters  worked  per  year  relative  to  the  control  group  and  these  latter  impacts  es¬ 
timates  are  no  longer  significantly  different  from  zero.25  The  employment  effects  of  the  GAIN 
programs  in  Alameda  and  San  Diego  also  decline  in  magnitude  and  statistical  significance  and 
the  impacts  attributable  to  GAIN  in  these  counties  remain  substantially  smaller  than  those  for 
Riverside.  However,  the  GAIN  impacts  on  the  two  measures  of  annual  employment  for  the  Los 
Angeles  program  grow  in  magnitude  in  years  4  through  9  relative  to  those  in  the  first  three  years. 
Recall  from  Table  1  that  the  GAIN  program  in  Los  Angeles  concentrated  its  services  on  long¬ 
term  welfare  recipients  at  the  time  our  sample  members  were  randomly  assigned  and,  from  Table 
4,  that  this  program,  at  that  time,  was  oriented  toward  the  providing  its  registrants  with  basic 
education  and  skill  development  programs.  On  average,  the  annual  employment  rates  of  the 
GAIN  registrants  in  Los  Angeles  are  3.3  (3.8)  percentage  points  greater  per  year  and  the  number 
of  quarters  worked  per  year  is  0.10  (1.3)  larger  than  the  corresponding  averages  for  control  group 
members  in  years  4  through  6  (years  7  through  9)  after  random  assignment.  These  later-year  im- 

25  We  also  note  that  the  average  employment  rates  and  quarters  worked  per  year  for  experimentals  in  Riverside  con¬ 
sistently  decline  in  magnitude  over  the  nine-year.  This  is  in  contrast  to  the  other  3  counties,  where  comparable  out- 
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pacts  impact  estimates  for  Los  Angeles  are  all  statistically  significant  and  are  larger  than  found 
for  any  of  the  other  three  county  programs,  including  Riverside’s. 

It  is  possible  that  the  larger  impacts  of  GAIN  on  employment  effects  found  in  Los  Ange¬ 
les  County  over  the  latter  three  years  of  our  post-enrollment  data  may  be  the  result  of  changes  in 
that  County’s  GAIN  program  that  were  initiated  in  1995.  (These  changes  were  in  effect  during 
years  6  through  9  of  our  analysis  period.)  In  1995,  Los  Angeles  County  re-oriented  its  GAIN 
program  toward  a  “work-first”  or  “job-first”  program,  adopting  a  program  model  similar  to  that 
used  in  Riverside.  All  members  of  our  sample  that  continued  to  reside  in  Los  Angeles  County 
and  remained  on  welfare — as  well  as  all  other  GAIN-mandated  adults  in  the  County’s  pro¬ 
gram — would  have  been  eligible  for  this  new  program  during  years  6  through  9.  Moreover,  re¬ 
cent  evidence  from  a  random-assignment  evaluation  of  the  2-year  post-enrollment  impacts  of 
Los  Angeles’s  re-oriented  GAIN  program  indicate  that  it  had  positive  employment  effects  on 
AFDC-FG  adults,  similar  to  the  initial  effects  found  for  the  Riverside  program.  Consistent  with 
the  possible  impacts  of  Los  Angeles’s  reoriented  program  is  the  fact  that  the  employment  rates 
and  quarters  worked  for  both  experimental  and  control  group  members  in  our  sample  increased 
in  years  6  through  9,  relative  to  earlier  years.  While  we  cannot  rule  out  this  explanation  for  the 
larger  employment  effects  in  Los  Angeles,  we  find  little  evidence  that  the  change  in  the  Los  An¬ 
geles  GAIN  program  had  noticeable  affects  on  the  other  outcomes  (earnings  and  welfare  partici¬ 
pation)  in  years  6  through  9.  It  would  also  be  difficult  to  explain  in  this  interpretation  why  the 
gain  in  the  later  years  is  larger  for  the  experimental  group  than  the  control  group.  (More  on  this 
below.) 


comes  for  experimental  in  each  of  the  other  three  counties  increased  over  the  nine-year  follow-up  period. 

26  Freedman,  et  al.  (1999,  2000)  provide  the  official  results  from  MDRC’s  evaluation  of  Los  Angeles  Job-First 
GAIN  program. 
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The  impacts  of  GAIN  programs  on  earnings  and  our  indicator  of  poverty  for  AFDC-FG 
households  are  displayed  in  Table  5.  As  with  the  impacts  on  employment,  we  find  that  the  differ¬ 
ences  in  earnings  and  in  the  incidence  of  earnings  being  greater  than  our  “threshold”  for  pov¬ 
erty— namely,  that  a  sample  member’s  annual  earnings  exceeded  the  income  generated  by  work¬ 
ing  full  time  at  the  minimum  wage — between  experimentals  and  controls  tends  to  decline  in  both 
Riverside  and  San  Diego  over  the  nine-year  follow-up  period.  In  the  case  of  Riverside,  annual 
earnings  gains  go  from  an  average  of  $1,416  per  year  in  the  first  three  years  to  an  annual  average 
of  $41 1  over  the  last  three  years.  The  comparable  averages  for  San  Diego  are  $616  and  $446,  re¬ 
spectively.  Nonetheless,  we  note  that  the  impacts  on  earnings  in  Riverside  are  sizeable  and  re¬ 
main  so,  even  six  to  nine  years  after  individuals  were  randomly  assigned  in  that  county.  With  re¬ 
spect  to  the  effects  of  the  GAIN  programs  in  Alameda  and  Los  Angeles  counties  on  earnings  and 
our  poverty  measure,  our  estimates  of  three-year  averages  over  the  nine-year  period  are  seldom 
statistically  significant,  although  we  do  find  that  the  magnitude  of  the  impacts  almost  always  in¬ 
crease  in  years  4  through  9  relative  to  those  for  years  1  through  3. 

We  also  present,  in  Table  5,  estimates  of  the  impacts  of  the  GAIN  programs  on  welfare 
participation  over  the  nine  years  after  random  assignment  for  AFDC-FG  GAIN  registrants.  As  is 
clear  from  the  estimates  in  this  panel,  the  GAIN  participants  in  each  of  the  counties  consistently 
have  lower  rates  and  quarters  of  welfare  participation  than  their  control  group  counterparts  over 
the  nine-year  period  and  these  differences  are  statistically  significant  in  many  of  the  years  after 
random  assignment,  including  the  latter  four  years.  Clearly,  the  welfare  reductions  are  largest  for 
Riverside,  with  GAIN  registrants  who  averaged  a  5.8  percentage  point  average  annual  lower  rate 
of  AFDC/TANF  participation  than  the  control  group  in  the  first  three  years  after  random  assign¬ 
ment  and  a  4.8  (3.2)  percentage  point  differential  in  years  4  through  6  (years  7  through  9).  While 
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the  welfare  reductions  attributable  to  the  GAIN  program  in  San  Diego  are  smaller  in  magnitude 
than  in  Riverside,  the  effects  for  this  county  also  are  statistically  significant  in  almost  every  year. 
Finally,  while  the  GAIN  registrants  in  Alameda  and  Los  Angeles  GAIN  programs  also  experi¬ 
enced  evidence  of  welfare  reductions,  the  effects  in  these  two  counties  tended  to  be  smaller  in 
magnitude  and  less  reliably  estimated,  especially  in  the  last  two  years  of  the  follow-up  period. 

In  Table  6,  we  present  the  corresponding  estimates  of  long-term  impacts  for  the  GAIN 
registrants  for  the  AFDC-U  cases  in  Alameda,  Los  Angeles,  Riverside  and  San  Diego  counties. 
There  are  several  notable  differences  in  the  findings  for  two-parent  AFDC  households  compared 
to  those  found  for  single-parent  (and  largely  female-headed)  AFDC  households  recorded  in  Ta¬ 
ble  5.  With  respect  to  the  impacts  on  employment,  most  of  the  gains  for  GAIN  registrants  in  each 
of  these  counties,  at  least  the  ones  that  are  precisely  estimated,  occur  in  the  first  six  years  after 
random  assignment.  Second,  the  GAIN  program  in  Los  Angeles,  rather  than  those  in  Riverside 
or  San  Diego  counties,  shows  the  largest  impacts  during  the  first  six  years  after  random  assign¬ 
ment,27  although  the  impacts  in  this  county  fall  off  markedly  after  7  and  9  years  after  random  as¬ 
signment.  Turning  next  to  the  GAIN  impacts  on  earnings  and  poverty  for  AFDC-U  households 
in  Table  6,  we  find  that  virtually  none  of  the  annual  impact  estimates  are  precisely  estimated  for 
any  of  the  years.  It  is  notable,  however,  that  the  impacts  on  earnings  for  GAIN  registrants  in 
Alameda  county  are  substantially  larger  in  years  4  through  9 — an  average  impact  of  $654  per 
year  in  years  4  through  6  and  $935  in  7  through  9 — relative  to  those  for  the  first  three  years — an 
average  impact  of  $54  per  year — although  none  of  these  estimates  are  statistically  significant  at 
conventional  levels  of  significance. 

Finally,  the  results  in  Table  6  for  welfare  participation  show  that,  for  AFDC-U,  the  most 

27  Freedman,  et  al.  (1996)  note  this  in  their  5-Year  GAIN  Impact  Analysis  working  paper. 
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persistent  reductions  in  welfare  over  the  nine  years  following  random  assignment  occur  for  the 
GAIN  participants  in  Alameda  and  Los  Angeles  counties,  the  two  counties  that  emphasized  basic 
education  and  skill  development  in  their  programs.  Moreover,  the  reductions  in  welfare  depend¬ 
ence  actually  improve  over  time  for  these  two  counties.  For  example,  the  GAIN  registrants  in 
Alameda  had,  on  average,  a  6.9  percentage  point  lower  rate  of  participation  in  AFDC/TANF  than 
their  control  group  counterparts  in  the  first  three  years  after  random  assignment  and  a  17.2  (13.9) 
percentage  point  lower  rate  in  years  4  through  6  (years  6  through  9).  While  the  reductions  for 
GAIN  registrants  are  lower  in  Los  Angeles  County,  they  do  ‘  improve  from  a  2.7  percentage 
point  reduction  in  the  first  three  years  to  an  average  reduction  of  5.0  and  3.3  percentage  points 
per  year  in  years  4  through  6  and  7  through  9,  respectively. 

In  summary,  our  examination  of  the  long-term  experimental  estimates  of  the  impacts  of 
the  GAIN  programs  in  these  four  counties  indicate  some  noticeable  differences  between  experi¬ 
mental  effects  in  the  years  immediately  following  random  assignment  (years  1  through  3)  com¬ 
pared  to  experimental  effects  at  longer  intervals  after  randomization  (years  7  through  9).  Fur¬ 
thermore,  the  results  on  the  longer  run  impacts  of  GAIN  are  less  supportive  of  the  view  that  the 
Riverside  GAIN  program  dominates  those  in  the  other  counties  we  analyze.  However,  drawing 
the  latter  conclusion,  while  tempting,  is  subject  to  the  flaw  that  was  noted  in  the  Introduction, 
namely,  that  the  experimental  design  does  not  support  conclusions  about  the  differential  effects 
of  programs  based  solely  on  evidence  from  within-county  random  assignment  evaluations.  In  the 
next  section,  we  discuss  an  econometric  strategy  to  get  at  these  differential  effects  and  discuss 
how  to  validate  its  reliability,  using  data  for  the  within-county  control  groups. 

4.  Identifying  the  Differential  Effects  of  GAIN  Programs 

In  this  section,  we  discuss  how  to  estimate,  or  identify,  the  differential  effects  of  treat- 
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ments.  As  noted  above,  the  results  from  the  MDRC  GAIN  Evaluation  have  been  interpreted  as 
suggesting  that  the  work-first  oriented  program  used  in  Riverside  County  is  relatively  more  ef¬ 
fective  than  the  human-capital  development  programs  used  in  the  other  counties,  especially 
Alameda  and  Los  Angeles  Counties.  Implicitly,  the  MDRC  GAIN  results  have  been  interpreted 
as  if  they  answered  the  following  question:  What  would  the  average  outcome  have  been  for  par¬ 
ticipants  in,  say,  Los  Angeles,  had  they  been  subject  to  the  program  as  implemented  in  River¬ 
side?  Clearly,  answering  this  question  is  of  crucial  important  to  the  administrators  in  these  coun¬ 
ties  and  the  rest  of  the  nation  for  deciding  which  welfare-to-work  model  to  implement. 

To  answer  this  question,  one  needs  to  determine  the  differential  effects  of  the  work-first 
oriented  treatment  used  in  Riverside  relative  to  the  human-capital  oriented  treatments  in  the  other 
counties.  As  we  have  suggested  in  the  Introduction,  the  within-county  experimental  design  em¬ 
ployed  in  the  GAIN  Evaluation  does  not  immediately  lend  itself  to  estimating  differential  effects. 
The  within-site  experimental  design  used  in  the  MDRC  GAIN  evaluation  only  allows  compari¬ 
sons  of  average  sites  against  an  average  of  alternative  sites,  rather  than  a  comparison  of  two  spe¬ 
cific  sites  against  each  other,  as  we  attempt  to  do  here.  Randomization  of  treatments  across  sites 
would  be  required  to  obtain  experimental  estimates  of  the  differential  effects  of  these  treatments. 
Without  randomization  over  sites  the  problems  in  comparing  program  results  in  different  loca¬ 
tions  are  similar  to,  although  distinct  from,  those  in  justifying  a  causal  interpretation  of  treat¬ 
ment-control  differences  in  non-experimental  evaluations.  Below,  we  characterize  more  pre¬ 
cisely  why  this  is  so. 

In  an  effort  to  obtain  estimates  of  differential  impacts  using  data  on  cross-county  com¬ 
parisons,  we  outline  how  matching,  or  regression-adjustment,  estimation  methods  could  be  used 
to  estimate  the  differential  effects  of  alternative  program  treatments,  such  as  work-first  versus 
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human-capital  development  ones  and  the  conditions  required  for  one  to  use  such  methods  with 
the  four-county  data  available  in  the  MDRC  Evaluation.  While  such  methods  are  inherently  more 
controversial  than  those  from  a  properly  designed  experiment  in  which  different  treatments  are 
randomly  assigned  to  subjects  from  a  particular,  population,  recent  studies  by  Dehejia  and  Wa- 
hba  (1999),  and  Heckman,  Ichimura,  Smith  and  Todd  (1997,  1998a)  suggest  such  adjustments 
with  sufficiently  detailed  observable  characteristics  may  lead  to  credible  non-experimental  esti¬ 
mates  of  average  gross  treatment  effects  (AGTEs).  Herein,  we  examine  whether  the  effectiveness 
of  these  methods  extend  to  estimating  the  differential  effects  of  alternative  treatments.  In  our  as¬ 
sessment,  we  discuss  how  one  can  use  the  data  on  the  experimentally-generated  control  groups 
for  these  four  counties  to  validate,  in  part,  these  conditions  for  use  of  regression-adjustment  and 
matching  methods. 

Recalling  the  discussion  in  Section  3.1,  we  now  consider  the  identification  of  differential 
treatment  effects.  Adopting  the  notation  used  in  that  section,  we  first  define  the  overage  differ¬ 
ential  treatment  effect  (ADTE)  of  treatment  k  relative  to  treatment  If  as, 

r,  (*,  *')  *  E(Y„  (*)  -  Y„  (k'))  =  E  (Art  (k)  -  A,  (*')) ,  (4) 

where  the  second  equality  in  (4)  follows  from  the  definition  of  A ,-,(/).  The  conditional  analogue  of 
(4)  is  given  by 

for  k  and  If  e  {w,h},  k^  If . 

There  are  at  least  two  reasons  why  data  from  the  MDRC  GAIN  county-specific  experi¬ 
mental  evaluations  do  not  necessarily  allow  us  to  identify  ADTEs.  First,  as  noted  above,  the 
treatments  received  by  households  varied  by  their  county  of  residence,  with  Riverside’s  GAIN 
program  emphasizing  work-first  treatments  (T=w)  while  the  programs  in  Alameda  and  Los  An- 
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geles  Counties  emphasized  human-capital  development  treatments  (T-  h).2S  More  formally,  this 
implies  that 

T,£Dt.  (C-2) 

In  an  extreme,  we  might  consider  7}  is  a  deterministic  function  of  A,  i*e., 

Pr(7;  =  kd  |  A  =d)  =  1  and  Pr(7;  =  k'd.  \D,  =  d)  =  0,  (C-2') 

for  all  kd,kd,  ^  0.  This  characterization  of  the  dependence  between  T  and  D  seems  reasonable 

since,  as  noted  above,  Riverside’s  entire  GAIN  program  was  work-first  oriented  while  the  GAIN 
programs  in  Alameda  and  Los  Angeles  Counties  were  oriented  toward  human  capital  develop¬ 
ment,  at  least  over  the  periods  of  enrollment  in  the  MDRC  GAIN  Evaluation.  Throughout  the 
remainder  of  this  paper  we  will  assume  that  condition  (C-2')  holds.  Second,  the  populations — 
and  their  potential  incomes,  Yt(k) — may  differ  across  counties,  i.e., 

Y„(k)£D„  (6) 

for  all  k  e  (0,  w,  h).  As  a  consequence,  there  is  no  guarantee  that  the  average  differences  in  out¬ 
comes  for  treatments  kd  and  kd, ,  specific  to  two  counties  d  and  d,  respectively,  will  identify  the 

ADTE  for  these  two  treatments.  To  see  this,  we  write  the  potential  outcome  associated  with 
treatment  k  that  is  received  in  County  d  as  follows: 

Y(kd)  =  Yd  (0)  +  Ad (k) ,  (7) 

where  both  I'f(O)  and  A d(k)  are  county-specific  and  are  treated  as  random  variables.  From  the 
available  data,  one  can  identify  the  differences  in  mean  outcomes  for  those  receiving  treatments 
kd  and  kd., 

E(Y*{kd)\T,  =  kd,D,  =  d)-E(Yf(kd.)\Tl  =  kfd.,D,  =  d') , 

28  San  Diego’s  GAIN  program  was  a  mixture  of  these  two  types  of  programs. 
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across  Counties  d  and  d.  Using  the  characterization  of  the  outcomes  associated  with  these  treat¬ 
ments  in  (7),  it  follows  that 

E(r„(K)\T,  =*„A  =d)-E(W,)\r,  =*;.A =<*') 

=  £(i;,'(0)  +  A^(t,)|7;  =k„.D,  =d)-E(rf<(>)  +  &UK')\T,=k:,.,D1  =d') 

1  (8) 

={E(AUkd)\T,  =  kd,Di  =d)-E(AUO\T,=k'„D,  =<*')} 

+{£(!',“ '(0)|t;  =*„  A  =rf)-£(n‘ r  (0)|7; = t;,A  =<*')}• 

In  general,  the  expression  in  (8)  is  not  equal  to  (4),  the  ADTE  for  treatments  k  and  k! .  Their 
equality  requires  that  the  following  two  additional  conditions  hold: 

£(i?(0)|7;  =  kd,Dt  =d)-E(YZ\G)\Ti  =  £,/>,  =^)  =  °,  (A-l) 

for  all  d,  d  and  t  i.e.,  there  is  no  difference  in  the  no-treatment  outcomes  across  counties  d  and 

d,  and 

£(A'(*))  =  £(A, .,(*»,  (A'2) 

for  all  d  and  /,  i.e.,  the  expected  gross  treatment  effect  of  treatment  k  does  not  vary  with  the 

county  of  residence  d. 

The  within-county  experimental  design  employed  in  MDRC’s  evaluation  of  the  GAIN 
programs  in  its  analysis  counties  does  not  guarantee  that  conditions  (A-l)  and  (A-2)  hold.  More¬ 
over,  the  descriptions  of  the  GAIN  registrants  and  programs  across  the  four  counties  presented  in 
Section  2  do  not  lend  support  to  either  of  these  conditions.  Recall  the  pre-enrollment  differences 
in  earnings,  labor  force  participation  and  welfare  receipt  between  the  GAIN  registrants  in  River¬ 
side  relative  to  those  in  Alameda  or  Los  Angeles  Counties  and  the  differences  in  personal  char¬ 
acteristics  (Table  1)  and  the  differences  in  labor  market  conditions  across  counties  at  the  time  of 
enrollment  into  the  MDRC  GAIN  evaluation  (Table  2).  These  differences  suggest  that  (A-l)  will 
not  hold,  i.e.,  that  there  are  differences  in  the  populations,  and  thus,  the  T}((0)s,  across  counties, 
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irrespective  of  county  differences  in  GAIN  programs.  Condition  (A-2)  stipulates  that  the  gross 
gains  for  the  same  treatment  must  be  the  same  in  all  counties.  If  the  average  gross  gain  from  a 
treatment,  such  as  human  capital  development,  depends  on  a  registrant’s  educational  attainment 
and  language  skills  and  there  are  differences  in  the  distribution  of  these  initial  skills  in  the  regis¬ 
trant  population  differ  across  counties,  Condition  (A-2)  will  be  violated.  Alternatively,  if  the  ef¬ 
fectiveness  of  a  particular  treatment,  such  as  work-first  treatments,  depends  on  economic  condi¬ 
tions,  i.e.,  the  availability  of  jobs  in  a  labor  market,  Condition  (A-2)  would  be  violated.  Again, 
the  differences  in  educational  backgrounds  of  registrants  and  labor  market  conditions  across  the 
four  counties  in  the  MDRC  GAIN  Evaluation  discussed  in  Section  2  suggest  that  (A-2)  is 
unlikely  to  hold  in  our  data. 

To  deal  with  the  lack  of  credibility  in  maintaining  (A-l)  and  (A-2)  for  the  GAIN  evalua- 
tion  data,  we  consider  the  use  of  matching  and/or  regression-adjustment  methods.  Such  meth¬ 
ods  are  predicated  on  the  availability  of  a  sufficiently  rich  set  of  observable  background  charac¬ 
teristics  and  outcome  histories  for  GAIN  registrants  and  measures  of  labor  market  conditions  that 
can  be  used  to  adjust  for  the  differences  across  counties  so  that  conditional  versions  of  (A-l)  and 
(A-2)  hold.  More  precisely,  while  (A-l)  and  (A-2)  need  not  hold,  suppose  that  the  following 
conditional  versions  of  them  do: 

£((tf(0)|7;  =  *,A  =d)-E(YfQS)\T,  =*;,A  =OIX>)  =  0  (A-l') 

and 

E  ( E(  At  (*))  -  E  ( Af  (k))\Xt )  =  0 ,  (A-2') 

for  all  k  e  (w,  h).  Assumptions  (A-l')  and  (A-2')  are  sufficient  to  justify  the  use  of  non-paramet- 

29  See  Rubin  (1979)  and  Heckman,  LaLonde  and  Smith  (1999)  for  discussions  of  these  methods  and  the  conditions 
that  support  them. 
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ric  methods  and,  in  certain  cases,  parametric  regression  techniques  to  identify  the  ADTEs  in  (4) 
and  (5).30  By  matching  individuals  in  one  county  with  comparable  individuals  (based  on  A)  in  the 
other,  one  can  eliminate  both  the  population  differences  that  exist  in  the  absence  of  any  treat¬ 
ments  as  well  as  the  county-specific  factors  that  lead  to  differences  in  gross  gams  associated  with 
a  particular  treatment  across  counties. 

As  with  any  non-experimental  methodology,  maintaining  assumptions  like  (A-l')  and  (A- 
2')  are  inherently  controversial,  because  their  validity  typically  cannot  be  ensured  or  verified. 
Note,  however,  that  in  our  case  the  validity  of  (A-l')  can  be  examined  empirically,  given  that  we 
have  data  on  controls  from  the  within-county  experiments  conducted  by  MDRC.  In  particular. 
Condition  (C-l'),  implies  that 

E(Y^^)\Ti=kd,Di  =  d)  =  £(#  (0)|7;  =0,  D,  =d)  (C-l') 

for  all  d  and,  thus,  that  the  mean  outcomes  for  control  group  members  in  each  county  is  a  con¬ 
sistent  estimator  of  E(Y*(0)\Ti  =  kd,Di  =  d).  (Note  that  this  result  holds  for  observable  sub¬ 
groups  of  the  GAIN  registrant  populations  as  well.  As  a  result,  we  can  test  whether  Assumption 
(A-l')  holds  in  our  data.  (A  similar  test  for  Assumption  (A-2')  cannot  be  performed.  Thus,  this 
assumption  will  need  to  be  maintained.  However,  in  our  view,  maintaining  it  is  much  less  con¬ 
troversial  so  long  as  (A-l)  holds.)  Furthermore,  it  follows  from  (C-10  and  (8)  that  one  can  use 
the  data  on  expected  outcomes  for  control  group  members  in  counties  d  and  cT  to  identify  ADTE 
for  the  treatments,  kd  and  k',  used  in  these  two  counties,  so  long  as  Assumption  (A-2')  holds. 


30  Stronger  forms  of  these  conditions  are  often  invoked  in  the  matching  literature.  In  particular,  in  place  of  (A-l  ) 
and  (A-2'),  the  following  conditional  independence  assumptions  are  often  used: 


and 


A„(*)±Z),k,- 


(A-l") 

(A-2") 
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More  formally,  (C-l')  and  (8)  imply  that 

[E(Y,AK)\T,=K,D,  =  </)-£(#  (0)|7;  =0,D,  =rf)] 

-[e(TM)%  =k'd.,D,  =d')-E(rf  (0)|7-  =0,A  =rf')]  (9) 

=£(4;ft)|3’=*„A=rf)-£(Af(t;)|2;=t;>A=rf') 

Thus,  the  availability  of  the  within-county  experimental  data  implies  an  alternative  way  of  iden¬ 
tifying  the  ADTEs  for  these  two  treatments  so  long  as  (A-2)  holds. 

In  the  empirical  analysis  that  follows,  we  make  use  of  parametric  regression  methods, 
rather  than  non-parametric  matching  techniques,  to  condition  on  the  A”s  as  suggested  by  as¬ 
sumptions  (A- 1 ')  and  (A-2').31  We  focus  on  contrasting  the  effects  of  the  Riverside  work- first 
oriented  program  relative  to  those  for  the  more  human-capital  development  oriented  programs  in 
each  of  the  other  three  counties  (Alameda,  Riverside,  and  San  Diego).  For  one  of  these  contrasts, 
let  Ri  =  1  denote  a  registrant  being  enrolled  in  Riverside’s  program  and  0  for  those  residing  in  the 
comparison  county.  Let  T,  be  defined  here  as  equal  to  one  if  the  registrant  was  in  the  experimen¬ 
tal  group  in  a  particular  county  and  equal  to  zero  otherwise.  Below,  we  present  estimates  for  two 
alternative  regression  specifications.  In  the  first,  we  use  data  only  on  experimental  subjects  (T,  = 
1)  from  Riverside  and  a  comparison  county  to  estimate  the  following  regression  model: 

i ',  =  A,+M+fiLX,+e..  d°) 

where  eit  is  a  stochastic  disturbance  assumed  to  have  mean  zero.  The  coefficient  on  the  Riverside 
dummy  (/?],)  measures  the  effect  of  the  program  in  Riverside  relative  to  the  program  in  the  com¬ 
parison  county  (Los  Angeles,  Alameda,  or  San  Diego)  on  outcome  Y  as  of  period  t.  The  estimate 

For  our  purposes,  we  only  require  the  conditional  mean  independence  assumptions  in  (A-l')  and  (A-2'). 

31  While  not  presented  herein,  we  also  used  non-parametric  matching  techniques,  controlling  for  the  same  set  of  A’s 
listed  above,  to  estimate  the  differential  treatment  effects  between  Riverside  and  the  various  comparison  counties. 
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of  this  coefficient  provides  a  consistent  estimator  of  y,  in  (4)  if  assumptions  (A-l')  and  (A -2') 
hold.  In  estimating  this  linear  regression  function,  we  include  in  Xi  personal  characteristics  (indi¬ 
cator  for  female,  indicators  for  five  levels  of  education,  Hispanic,  black,  an  indicator  for  having 
one  child,  an  indicator  for  having  children  under  the  age  of  five),  past  earnings  and  labor  force 
participation  indicators  (for  quarters  one  to  ten  prior  to  randomization),  indicator  for  past  receipt 
of  welfare  and  the  amounts  of  AFDC  benefits  received  (for  quarters  one  to  four  prior  to  ran¬ 
domization),  for  the  head  of  household  each  household,  as  well  as  variables  describing  individual 
labor  market  histories  at  the  time  of  random  assignment.  We  refer  to  the  estimates  of  fi\,  below 
as  the  “regression-adjusted  difference  in  means  for  experimentals.”  We  also  present  estimates  fi\t 
for  a  variant  of  (10)  that  does  not  include  the  X  in  order  to  depict  the  consequences  of  control¬ 
ling  for  these  observables.  Below,  we  refer  to  the  estimates  for  this  variant  as  the  “unadjusted 
difference  in  means  for  experimentals.” 

Estimating  the  specification  in  (10)  (that  includes  X)  with  data  for  members  of  the  experi¬ 
mental  groups  from  Riverside  and  a  comparison  county  presumes  that  both  conditions  (A-l')  and 
(A-2')  holds.  As  noted  above,  the  availability  of  county-specific  controls  groups  that  were  deter¬ 
mined  by  random  assignment,  allows  us  scope  for  testing  the  validity  of  (A-l').  To  test  this  as¬ 
sumption,  we  estimate  the  specification  in  (10)  using  data  for  members  of  the  controls  groups  of 
Riverside  and  the  comparison  county.  Given  this  data  on  control  groups,  condition  (A-l')  implies 
that  {5\t  —  0  in  (10).  That  is,  if  our  regression-adjustment  method  is  successful,  there  should  there¬ 
fore  be  no  difference  in  average  outcomes  of  individuals  in  the  control  groups  between  the  sites 
and  the  estimate  for  f5\  should  be  close  to  zero,  both  substantively  and  statistically.32  Below,  we 


The  estimates,  especially  the  inferences  drawn,  are  quite  similar  to  the  regression-based  estimates  reported  below. 

32  For  other  examples  of  evaluating  procedures  by  applying  them  to  groups  for  whom  the  effects  are  known  (typi- 
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present  estimates  of  fix,  for  the  various  outcomes  of  control  group  members  for  each  of  the  Riv¬ 
erside-comparison  County  contrasts  for  specifications  of  (10)  that  include  and  do  not  included. 
The  resulting  estimates  of  f3\,  that  do  not  control  for  X are  referred  to  as  “unadjusted  difference  in 
means  for  controls”  and  those  that  do  are  labeled  as  “regression-adjusted  difference  in  means  for 
controls.” 

Failure  to  reject  the  null  hypothesis  that  fi\t  =  0  for  outcomes  of  the  controls  groups  in 
Riverside  and  a  comparison  county  even  though  the  null  is  not  true,  i.e.,  a  Type  II  error  may  be 
committed  in  testing  fix,  =  0.  To  guard  against  the  consequences  of  this  type  of  error  while,  at  the 
same  time,  exploiting  the  availability  of  data  on  randomly  generated  control  groups  within  each 
of  the  counties,  we  also  present  estimates  for  two  “difference-in-differences”  estimators.  The 
first  is  the  sample  analogue  of  (9),  i.e.,  the  difference-in-differences  of  the  mean  outcomes  for 
experimental  and  controls  for  Riverside  and  the  comparison  county.  This  estimate  corresponds 
to  the  coefficient,  flit,  in  the  following  regression: 

Y„  =  P0l  +  M  +  P*Ti  +  PM  +  V  (11) 

This  estimator  is  a  consistent  estimator  for  the  ADTE  for  the  treatments  used  in  these  two  coun¬ 
ties  so  long  as  condition  (A-2)  holds.  We  refer  to  the  estimates  for  this  estimator  in  the  tables  be¬ 
low  as  the  “unadjusted  difference-in-differences.”  We  also  present  a  difference-in-difference  es¬ 
timator  of  the  ADTEs  that  controls  for  X,  i.e., 

r„ = A,  +M  +M +mt,+/};,x,  +fcx,T,  +v„,  02) 

where  the  /%»  corresponds  to  y.  Below,  we  refer  to  estimates  of  A/  as  the  “regression  adjusted 
difference-in-differences”  estimates. 


cally  zero,  as  in  this  case),  see  Lalonde  (1986),  Heckman  and  Hotz  (1989),  Rosenbaum  (1995),  Hotz,  Imbens  and 
Mortimer  ( 1 999). 
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Because  the  latter  difference-in-differences  estimator  in  (12)  (flit)  exploits  the  experimen¬ 
tal  data  on  control  groups  and  only  relies  on  condition  (A-2)  holding,  it  is  arguably  a  more  credi¬ 
ble  non-experimental  estimator  of  the  differential  effects  of  the  alternative  treatments  used  in 
Riverside  and  the  comparison  counties.  In  essence,  this  estimator  eliminates  the  additive  differ¬ 
ences  between  Riverside  and  the  other  sites  (via  the  control  group  outcomes)  as  well  as  adjusts 
for  the  observable  Xs.  It  should  be  kept  in  mind,  however,  that  if  the  differences  in  average  out¬ 
comes  for  controls  between  Riverside  and  Los  Angeles  (or  Alameda  or  San  Diego)  are  elimi¬ 
nated  by  adjusting  for  pre-randomization  variables — that  is,  if  the  coefficient  on  the  Riverside 
dummy  is  close  to  zero  in  the  control  regression — the  trainee-only  estimates  should  be  close  to 
the  difference-in-differences  estimates  (but  the  former  estimated  more  precisely  than  the  latter). 
If,  on  the  other  hand,  the  control  estimates  are  far  from  zero,  one  might  be  concerned  that  there  is 
some  important  unobserved  effect  on  outcomes  or  that  the  differences  between  Riverside  and  the 
other  sites  are  not  necessarily  additive,  and  the  difference-in-differences  estimates  would  be  less 
credible.  The  second  concern  with  the  difference-in-differences  estimates  is  that  the  statistical 
significance  levels  may  be  affected  by  the  relative  scarcity  of  the  control  groups.  For  example,  in 
Riverside  there  are  4,358  trainees,  but  only  1,025  controls.  If  we  can  successfully  control  for 
cross-county  differences  in  the  control  group,  then  we  may  prefer  the  adjusted  difference  for  the 
trainees  in  a  bias/variance  tradeoff  similar  to  the  one  concerning  the  decision  to  include  addi¬ 
tional  controls  with  limited  explanatory  power  in  a  linear  regression  framework. 

5.  Estimates  of  Differential  Effects  of  GAIN  Programs 

In  this  section,  we  present  estimates  of  the  differential  effects  of  the  GAIN  programs  run 
in  Riverside  County,  relative  to  those  in  Alameda,  Los  Angeles  and  San  Diego  counties.  Results 
for  AFDC-FG  cases  are  presented  in  Table  7  and  those  for  AFDC-U  cases  in  Table  8.  The  Ap- 
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pendix  presents  results  separately  for  the  in  need  of  basic  education  and  not  in  need  of  basic  edu¬ 
cation  groups.  In  what  follows,  we  discuss  the  estimates  for  sets  of  differential  effects,  namely 
Riverside  vs.  Alameda,  Riverside  vs.  Los  Angeles,  and  Riverside  vs.  San  Diego. 

5.1  Riverside  versus  Alameda 

The  results  in  the  left  set  of  columns  in  Table  7  presents  the  results  for  the  Riverside-Ala- 
meda  comparison  for  the  AFDC-FG  group.  The  first  set  of  results  presents  estimates  for  the  dif¬ 
ferences  between  Riverside  and  Alameda  for  yearly  employment  indicators.  The  simple  differ¬ 
ence  in  outcomes  for  trainees  averaged  over  the  first  three  post-randomization  years  shows  that 
1 8.2%  more  trainees  in  Riverside  are  employed  than  trainees  in  Alameda.  However,  even  the  dif¬ 
ference  between  Riverside  and  Alameda  for  controls,  7.3%  is  significantly  different  from  zero  (at 
the  1%  level).  Consistent  with  the  descriptive  results  on  differences  in  the  enrolled  population 
across  counties,  this  suggests  that  the  populations  enrolled  in  GAIN  differed,  so  simply  compar¬ 
ing  outcomes  for  the  trainees  is  not  appropriate.  The  difference-in-differences  estimate,  equal  to 
the  difference  between  the  trainee  and  control  differences  is  10.9%. 

Allowing  for  heterogeneity,  we  include  covariates  in  the  levels  and  as  interactions  with 
the  treatment  effects  to  estimate  the  net  effect  of  the  Riverside  GAIN  program  relative  to  the 
Alameda.  Adjusting  for  covariates,  the  estimated  differences  between  Riverside  and  Alameda 
averaged  over  the  first  three  years  changes  to  16.0%  for  trainees  and  3.7%  for  controls.  Note  that 
the  difference  between  Riverside  and  Alameda  for  controls  is  halved,  and  is  no  longer  signifi¬ 
cantly  different  from  zero  even  at  the  10%  level.  The  point  estimate  of  the  net  effect  of  Riverside 
relative  to  Alameda  increases  from  10.9%  to  12.3%. 

For  the  other  two  time  periods,  years  4-6  and  years  7-9,  the  differences  between  trainees 
in  Riverside  and  Alameda  are  substantial,  and  many  cases  significantly  different  from  zero.  Ad- 
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justing  for  pre-randomization  differences  does  not  affect  these  estimates  much.  For  the  controls 
some  of  the  raw  differences  between  Riverside  and  Alameda  are  large  and  significant,  but  ad¬ 
justing  for  pre-randomization  differences  makes  these  differences  substantially  smaller  and 
largely  insignificant.  Unlike  the  first  three  years,  the  adjusted  difference-in-difference  estimates 
are  not  significantly  different  from  zero  in  the  last  six  years.  The  estimated  net  effect  declines 
from  12.3%  for  the  first  three  years  to  5.2%  for  years  4-6  and  -0.4%  for  years  7-9. 

For  the  second  outcome,  the  number  of  quarters  employed  in  a  year,  the  same  pattern 
emerges.  The  raw  differences  for  control  are  often  significantly  different  from  zero,  but  adjusting 
for  pre-randomization  characteristics  eliminates  a  substantial  part  of  these  differences  and  ren¬ 
ders  them  largely  insignificant.  The  trainee  differences  remain  robust  to  adjusting  for  covariates. 
The  difference-in-differences  estimates  follow  the  same  pattern  of  significant  and  positive  effects 
in  the  first  three-year  period  (0.43),  going  down  in  years  4-6  (to  0.24  years)  and  becoming  nega¬ 
tive  in  years  7-9  (-0.01). 

The  third  outcome  is  total  yearly  earnings.  The  general  pattern  is  repeated  again.  Control 
differences  between  Riverside  and  Alameda  are  substantial  and  significant  before  adjusting  for 
covariates,  but  much  smaller  and  all  insignificant  after  adjustment.  For  the  three  time  periods  the 
unadjusted  differences  are  $403,  $-141,  and  $-1,032,  which  reduces  after  adjusting  for  covariates 
to  $107,  $88  and  $-185.  Again  the  trainee  differences  are  more  robust.  In  this  case  the  pattern  of 
initial  positive  relative  effects  for  Riverside  followed  by  later  negative  relative  effects  is  re¬ 
peated,  but  now  the  negative  effects  in  the  final  years  are  significant  at  the  5%  level.  The  differ¬ 
ence-in-differences  estimates  decline  sharply,  from  $876  in  the  first  period,  to  $210  in  the  sec¬ 
ond,  and  -$91 1  in  the  third  three-year  period. 

For  the  fourth  outcome,  the  indicator  for  earnings  above  the  full-time  minimum  wage 
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level,  the  story  is  again  similar,  but  the  effects  are  now  imprecisely  estimated.  There  are  differ¬ 
ences  for  the  unadjusted  controls,  but  adjusting  them  for  pre-randomization  differences  makes 
them  smaller  and  insignificant.  Differences  between  trainees  remain,  with  the  program  in  River¬ 
side  in  early  years  significantly  more  successful  than  in  Alameda  in  the  early  years.  In  the  later 
years,  there  is  a  reversal — Alameda’s  outcomes  are  better,  and  in  the  last  three  years  the  differ¬ 
ences  for  the  trainees  are  significantly  different  from  zero. 

The  story  for  positive  annual  AFDC/TANF  receipt  is  different.  Here  raw  differences  are 
very  large  and  significant  for  controls,  and  adjustment  for  covariates  does  not  entirely  eliminate 
them,  although  it  makes  these  differences  smaller  and  considerably  less  significant.  In  contrast  to 
the  earnings  measures,  the  differences  between  Riverside  and  Alameda  for  trainees  are  not  robust 
to  adjusting  for  covariates.  In  a  typical  three-year  period  the  raw  difference  is  on  the  order  of 
15%  lower  participation  in  Riverside,  but  after  adjusting  this  is  reduced  to  about  8%  lower  par¬ 
ticipation.  Difference-in-difference  estimates  still  follow  the  same  pattern  of  the  earnings  results 
of  an  early  advantage  for  Riverside  (a  bigger  reduction  in  AFDC  participation  rates)  followed  by 
sharp  decline,  with  the  effects  in  later  years  close  to  zero.  It  should  be  kept  in  mind  here  that,  in 
contrast  to  the  earnings  data,  where  we  have  ten  quarters  of  pre-randomization  outcomes  includ¬ 
ing  both  the  binary  indicators  and  the  amount,  we  have  for  AFDC/TANF  receipt  only  the  binary 
indicator,  but  not  the  amount,  and  only  for  four  pre-randomization  quarters.  Perhaps,  just  as  to 
adjust  for  employment  and  earnings,  we  need  detailed  histories  of  employment  and  earnings,  to 
successfully  adjust  for  AFDC/TANF  receipt  we  would  need  detailed  histories  of  receipt  and 
payment  amounts. 

For  the  last  outcome,  the  number  of  quarters  with  positive  AFDC  receipt  the  story  is 
similar  to  that  for  annual  AFDC  receipt.  Control  differences  are  largely  but  not  completely 
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eliminated  by  adjusting  for  covariates  (they  remain  significantly  different  from  zero).  The  differ- 
ence-in-differences  estimates  are,  other  than  in  the  first  three-year  period,  all  small  and  not  sig¬ 
nificantly  different  from  zero. 

5.2  Riverside  versus  Los  Angeles 

Next  we  discuss  the  Riverside  versus  Los  Angeles  comparison.  Starting  again  with  the 
annual  employment  indicators,  we  find  that  there  are  substantial  differences  between  controls  in 
Riverside  and  Los  Angeles,  ranging  from  4.7%  to  10.9%,  and  which  are  significant  at  the  1% 
level  in  all  three  time-periods.  Adjusting  for  pre-randomization  differences  lowers  the  adjusted 
difference  between  controls  across  the  two  counties,  now  ranging  from  0.7  to  4.4%,  and  signifi¬ 
cant  at  the  5%  level  in  only  one  period.  In  contrast,  the  trainee  differences  are  significant  both 
before  and  after  adjustment.  It  is  interesting  to  note  that  in  years  7-9  the  unadjusted  estimate  for 
the  trainee  differences  suggests  a  significantly  higher  employment  rate  in  Riverside,  whereas  the 
adjusted  estimate  suggests  a  significantly  lower  employment.  The  adjusted  difference-in-differ- 
ences  estimates  suggest  a  pattern  similar  to  that  in  the  Riverside-Alameda  comparisons,  initial 
differences  are  large  and  in  favor  of  Riverside,  followed  by  a  substantial  and  steady  decline, 
leading  to  relatively  employment  rates  that  are  higher  in  Los  Angeles  than  in  Riverside,  although 
not  significantly  so. 

The  second  outcome,  the  number  of  quarters  employed  each  year  follows  the  same  pat¬ 
tern.  There  are  highly  significant  differences  between  controls  in  Riverside  and  Los  Angeles 
prior  to  adjustment  (significant  at  the  1%  level  in  two  of  the  periods),  but  these  differences  are 
much  smaller  and  insignificant  after  adjustment  (none  significant  at  the  10%  level).  For  trainee 
differences  we  again  find  that  the  beneficial  effect  of  the  Riverside  program  relative  to  Los  An¬ 
geles  disappears  after  six  years  and  turns  into  a  significant  comparative  advantage  for  Los  An- 


39 


geles.  Difference-in-difference  estimates  are  positive  and  significant  in  early  years  and  negative 
but  not  significant  in  later  years. 

For  the  level  of  earnings  the  pattern  repeats.  Raw  control  differences  are  largely  elimi¬ 
nated  by  adjustment  for  pre-randomization  variables,  whereas  for  trainees  the  adjustment  shows 
in  later  years  significant  advantages  for  the  Los  Angeles  program  in  contrast  to  the  early  benefits 
of  the  Riverside  program.  For  the  final  eamings-based  measure,  an  indicator  for  earnings  above 
the  full-time  minimum  wage  level  the  raw  differences  for  the  controls  are  already  small  and 
largely  insignificant;  the  regression  adjustments  make  them  even  smaller  and  less  significant,  but 
also  eliminate  a  substantial  part  of  the  differences  for  trainees. 

For  the  two  AFDC  outcomes  the  results  for  the  Riverside-LA  comparison  are  somewhat 
different  from  those  for  the  Riverside-Alameda  comparison.  As  before,  we  do  find  large  differ¬ 
ence  in  unadjusted  differences  for  the  control  groups.  However,  the  least  squares  adjustment 
eliminates  virtually  all  of  the  differences.  Whereas  the  raw  differences  for  annual  AFDC  partici¬ 
pation  range  from  -12.8%  to  -7.2%  significantly  different  from  zero  at  the  1%  level  in  all  three 
periods,  after  adjusting  the  differences  range  from  -2.8%  to  0.3%,  none  significant  even  at  the 
10%  level.  The  difference-in-difference  estimates  suggest  a  significantly  bigger  decrease  in  Riv¬ 
erside  than  Los  Angeles  in  the  early  years,  with  an  insignificant  effect  in  later  years.  For  the 
number  of  quarters  spent  on  AFDC  the  story  is  similar.  The  raw  differences  for  controls  range 
from  -0.56  to  -0.31,  all  significant  at  the  1%  level,  and  the  adjusted  differences  range  from  -0.16 
to  -0.07,  with  two  significantly  different  from  zero  at  the  10%  level. 

5.3  Riverside  versus  San  Diego 

Finally  we  discuss  the  Riverside  versus  San  Diego  comparisons.  Here  we  have  more  dif¬ 
ficulty  eliminating  the  differences  between  average  control  outcomes  than  in  the  other  com- 
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parisons.  This  is  somewhat  surprising  given  that  before  adjustment,  the  population  in  San  Diego 
looks  more  comparable  to  that  in  Riverside  than  the  other  two  counties.  Adjustment  does  make 
the  differences  smaller,  but  some  do  remain  significantly  different  from  zero. 

However,  for  the  two  employment  measures  the  general  pattern  of  the  coefficients  is 
similar  to  that  for  the  other  two  comparisons.  The  difference-in-differences  estimates  are  much 
larger  in  the  first  few  years  but  then  decline  and  they  are  not  significantly  different  from  zero  in 
the  later  years.  Unlike  in  the  comparisons  with  Alameda  and  Los  Angeles  the  differences  are 
never  negative,  always  suggesting  a  benefit  from  attending  the  training  in  Riverside  relative  to 
San  Diego. 

For  earnings  the  raw  differences  between  controls  is  large  and  significant.  Furthermore, 
the  difference  is  not  removed  by  adjusting  for  covariates.  The  difference-in-differences  estimates 
are  all  positive  after  adjusting  for  covariates. 

The  two  AFDC  outcomes  follow  an  interesting  pattern.  The  raw  differences  for  controls 
are  all  small  and  insignificant.  This  does  not  change  if  we  adjust  for  pre-randomization  variables. 
However,  for  the  trainees  the  adjustment  does  make  a  considerable  difference.  The  size  of  the 
difference  and  the  significance  goes  up  substantially.  This  is  true  both  for  annual  AFDC  receipts 
and  for  the  number  of  quarters  with  positive  AFDC  receipts.  For  example,  for  the  number  of 
quarters  with  positive  AFDC  receipts  the  raw  differences  for  trainees  range  from  -0.15  to  -0.07. 
After  adjusting  for  pre-randomization  differences  the  range  is  -0.30  to  -0.19.  The  difference-in¬ 
difference  estimates  are  also  negative  in  all  three  periods  for  both  outcomes  after  adjusting,  and 
are  often  highly  significant. 

5.4  Conclusions  from  Cross-County  Comparisons 

Generally  we  find  that  there  are  substantial  and  significant  differences  in  raw  averages 
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between  sites,  for  all  three  comparisons,  and  all  six  outcomes  over  the  three  three-year  periods 
post-randomization.  Adjusting  for  a  rich  set  of  pre-randomization  variables  reduces  and  in  most 
cases  essentially  eliminates  these  differences  for  the  control  groups.  Exceptions  are  the 
AFDC/TANF  receipts  in  Alameda  County  and  some  of  the  earnings  outcomes  in  San  Diego 
County  where  substantial  differences  from  Riverside  County  remain  even  after  least  squares  ad¬ 
justment. 

The  ability  for  most  comparisons  and  outcomes  to  adjust  away  the  differences  for  con¬ 
trols  suggests  that  the  adjusted  differences  for  trainees  (and  the  adjusted  difference-in-differences 
estimates  which  therefore  are  close  to  the  trainee  differences)  can  be  interpreted  as  estimates  of 
the  causal  effect  of  the  Riverside  County  program  versus  the  three  others.  Generally  we  find  sub¬ 
stantial  positive  effects  of  the  Riverside  County  program  (increasing  employment  rates  and  earn¬ 
ings,  and  lowering  AFDC  receipts)  in  the  first  three  years  post-randomization;  the  period  covered 
by  the  MDRC  evaluations.  The  effects,  however,  taper  off  and  sometimes  becoming  significantly 
negative  in  the  last  two  three-year  periods  in  Alameda  County  and  Los  Angeles  County.  This  in¬ 
terpretation  is  consistent  with  the  effects  of  job  search  assistance  being  shorter  lived  than  the  ef¬ 
fects  of  basic  skills  training,  although  without  individual  level  data  on  the  nature  of  the  training  it 
is  difficult  to  further  investigate  this  interpretation.  It  is  also  largely  consistent  with  the  interpre¬ 
tation  that  after  the  embargo  ended,  the  control  groups  received  training.  This  seems  particularly 
likely  in  Riverside  County  were  the  program  was  nearly  universal  in  the  period  after  randomiza¬ 
tion  ended;  but  less  likely  in  the  other  three  counties  were  the  GAIN  programs  appear  to  have 
been  smaller  and  the  commitment  to  universal  participation  enrollment  was  weaker.  We  note, 
however,  that  such  delayed  treatment  for  the  controls  would  not  explain  the  sign  change  in  the 
Riverside-LA  and  Riverside-Alameda  comparisons  found  for  employment  and  earnings  out- 
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comes. 


The  results  underline  the  important,  but  discomforting,  conclusion  that  short-term  evalua¬ 
tion  of  training  programs  can  be  misleading.  The  relative  ranking  of  programs  is  not  stable  over 
time.  Simple  extrapolations  of  early  results  to  later  results  do  not  appear  to  be  possible.  The  rela¬ 
tion  of  short-term  results  to  longer-term  results  appears  to  vary  with  program  content  in  ways 
consistent  with  a  priori  expectations.  Thus,  despite  the  demands  of  policy  makers  for  quick  re¬ 
sults  with  which  to  design  new  legislation  and  programs,  there  may  be  no  substitute  for  long¬ 
term  and  costly  follow-up,  and  thus  for  program  design-evaluation-redesign  cycles  lasting  a  dec¬ 
ade  or  more.  Conventional  follow-up  periods  of  three,  or  even  five,  years  may  simply  be  too 
short. 

Finally,  let  us  return  to  the  Riverside  versus  Alameda  comparisons  for  AFDC  receipt  that 
was  one  of  the  most  problematic  comparisons.  For  these  outcomes,  adjusting  for  pre-randomiza¬ 
tion  differences  did  not  eliminate  differences  for  the  controls.  For  the  Riverside-LA  and  River- 
side-San  Diego  comparisons  covariance  adjustment  was  adequate.  To  gain  some  insight  into 
these  difficulties,  consider  the  fraction  on  AFDC  in  each  of  the  last  four  quarters  prior  to  ran¬ 
domization.  In  Riverside  these  fractions  are  63%,  64%,  65%,  and  77%  in  chronological  order. 
For  San  Diego  these  fractions  are  57%,  59%,  60%  and  71%,  very  comparable  to  the  Riverside 
numbers.  For  Los  Angeles  the  fractions  are  98%,  99%,  99%,  and  99%,  much  higher  than  in  Riv¬ 
erside  and  San  Diego.  Finally,  in  Alameda  the  numbers  are  97%,  97%,  97%,  and  98%,  again 
much  higher  than  in  Riverside  and  San  Diego.  These  differences  between  Riverside  and  San 
Diego  on  the  one  hand,  and  Los  Angeles  and  Alameda  on  the  other  hand,  reflect  the  focus  of  the 
GAIN  program  in  the  latter  two  counties  on  long-term  AFDC  recipients.  This  focus  was  a  result 
of  insufficient  funds  to  enroll  all  eligibles  given  the  high  per  case  cost  of  the  their  HCD  strategy. 
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This  difference  in  selection  rules  in  turn  is  reflected  in  the  large  raw  differences  post-randomiza¬ 
tion  between  Riverside  and  both  Los  Angeles  and  Alameda.  These  differences  are  in  fact  largest 
between  Riverside  and  Alameda,  and  the  adjustment  for  four  quarters  of  pre-randomization 
AFDC  indicators  does  not  appear  to  be  sufficient  to  eliminate  them,  although  it  does  reduce  them 
substantially. 

6.  Conclusions 

In  this  paper  we  analyze  data  from  the  GAIN  experimental  evaluations  of  job  training 
programs.  Whereas  previous  researchers  have  had  only  five  years  of  post-randomization  out¬ 
comes  available,  we  have  observations  on  nine  years  of  earnings  and  welfare  receipts  after  ran¬ 
domization.  This  allows  us  to  explore  the  long-term  effects  of  these  programs.  We  find  that  the 
early  superiority  of  the  Riverside  program  with  its  stress  on  job  search  assistance  rather  than  ba¬ 
sic  skills  training  is  lessened  over  time.  In  the  later  years  the  programs  in  counties,  such  as  Ala¬ 
meda  and  Los  Angeles,  are  doing  as  well  as,  or  even  slightly  better,  than  Riverside. 

We  also  make  a  case  that  credible  comparisons  can  be  made  between  the  countries.  Al¬ 
though  such  comparisons  cannot  be  justified  by  the  randomization  alone,  we  exploit  the  presence 
of  control  groups  to  validate  such  comparisons  between  trainees.  We  find  that  in  the  early  years 
the  program  in  Riverside  did  indeed  lead  to  better  outcomes,  although  the  relative  benefits  of  the 
Riverside  program  do  disappear  over  time.  Our  analyses  show  the  importance  of  having  detailed 
characteristics  of  the  individuals  even  in  randomized  experiments.  The  results  presented  here  are 
also  encouraging  for  the  ability  of  non-experimental  methods  to  reproduce  the  results  of  experi¬ 
mental  results,  if  enough  detailed  information  on  individual  characteristics  (e.g.,  histories  of  em¬ 
ployment,  earnings,  and  welfare  receipt)  is  available. 
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Table  1:  Background  Characteristics  and  Pre-Randomization  Histories 
of  GAIN  Evaluation  Participants 
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Table  2:  Annual  Labor  Market  Conditions  by  County,  1989-901 


Year 

Alameda 

Los 

Angeles 

Riverside 

San  Diego 

Annual  Unemployment  Rate 

1988 

4.8% 

5.2% 

6.5% 

4.3% 

1989 

4.4% 

5.5% 

6.8% 

3.9% 

1990 

3.9% 

5.9% 

6.3% 

4.7% 

Annual  Employment  Growth  Rates 

1988 

2.1% 

0.9% 

7.4% 

2.0% 

1989 

2.1% 

1.6% 

7.4% 

3.5% 

1990 

4.7% 

4.0% 

8.5% 

6.0% 

Annual  Real  Wage  Income  i 

per  Worker 

1988 

$21,661 

$23,070 

$17,328 

$18,820 

1989 

$21,569 

$22,833 

$17,453 

$18,817 

1990 

$21,752 

$22,957 

$17,775 

$18,919 

Annual  Growth  Rates  in  Real  Wage  Income  per  Worker 

■■ 

1.0% 

mmmmm  fe/i. 

-0.5% 

-1.8% 

1990 

WSm% 

-1.1% 

0.1% 

Share  of  Employment  in  Manufacturing 

1988 

0.112 

0.165 

■ 

1989 

0.113 

0.172 

1990 

0.113 

0.176 

0.088 

0.096 

Share  of  Employment  in  Service  Sector 

1988 

0.293 

0.333 

0.279 

0.289 

1989 

0.283 

0.326 

0.273 

0.279 

1990 

0.282 

0.324 

0.275 

0.279 

Share  of  Employment  in  Pul 

?lic  Administration  ! 

1988  ! 

0.191 

0.107 

0.155 

0.227 

1989 

0.193 

0.107 

0.156 

0.228 

1990 

0.193 

0.106 

0.157 

0.230 

‘Shaded  entries  denote  years  in  which  random  assignment  was  conducted  in  various 
counties. 
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Table  3:  Distribution  of  Average  Monthly  Participation  in  Various  GAIN  Activities1 2 


Yr.:Qtr. 

Job  Club  & 
Job  Search 
Activities 

All  Other 
Job  Search 
Activities 

Basic 

Education 

Program 

Vocational 

Training 

OJT 

PREP* 

Supported 

Work& 

Transitional 

Employment 

Alameda 

1988:Q3 

0% 

0% 

0% 

1988:Q4 

0% 

0% 

0% 

1989:Q1 

21% 

53% 

26% 

0% 

0% 

0% 

1989:Q2 

34% 

2% 

37% 

27% 

0% 

0% 

0% 

1989:Q3 

35% 

2% 

36% 

27% 

0% 

0% 

0% 

1989:Q4 

33% 

9% 

44% 

12% 

0% 

0% 

0% 

1990:Q1 

29% 

5% 

44% 

22% 

0% 

0% 

0% 

1990:Q2 

45% 

3% 

38% 

13% 

1% 

0% 

0% 

1990:Q3 

24% 

4% 

32% 

37% 

0% 

4% 

0% 

1990:04 

19% 

5% 

38% 

32% 

0% 

6% 

0% 

Los  Angeles 

1988:Q3 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

1988:Q4 

0% 

0% 

8% 

92% 

0% 

0% 

0% 

1989:Q1 

14% 

0% 

72% 

14% 

0% 

0% 

0% 

1989:Q2 

23% 

1% 

61% 

15% 

0% 

0% 

0% 

1989:Q3 

22% 

2% 

68% 

8% 

0% 

0% 

0% 

1989:Q4 

23% 

4% 

65% 

8% 

0% 

0% 

0% 

1990:Q1 

19% 

7% 

63% 

12% 

0% 

0% 

0% 

1990:Q2 

16% 

5% 

64% 

15% 

0% 

0% 

0% 

1990:Q3 

29% 

6% 

49% 

15% 

0% 

0% 

0% 

1990:04 

15% 

3% 

58% 

24% 

0% 

0% 

0% 

Riverside 

1988:Q3 

51% 

9% 

21% 

20% 

0% 

0% 

0% 

1988:Q4 

62% 

7% 

20% 

10% 

0% 

0% 

0% 

1989:Q1 

56% 

3% 

26% 

14% 

0% 

0% 

0% 

1989:Q2 

63% 

5% 

20% 

12% 

0% 

0% 

0% 

1989:Q3 

64% 

3% 

19% 

14% 

1% 

0% 

0% 

1989:Q4 

45% 

2% 

32% 

21% 

0% 

0% 

0% 

1990:Q1 

52% 

3% 

23% 

22% 

0% 

0% 

0% 

1990:Q2 

52% 

1% 

24% 

23% 

0% 

0% 

0% 

1990:Q3 

61% 

3% 

19% 

17% 

0% 

0% 

0% 

1990:04 

55% 

4% 

22% 

19% 

0% 

0% 

0% 

San  Diego 

1988:Q3 

41% 

1% 

28% 

28% 

1% 

2% 

0% 

1988:Q4 

45% 

1% 

30% 

22% 

1% 

2% 

0% 

1989:Q1 

41% 

1% 

30% 

24% 

2% 

2% 

0% 

1989:Q2 

42% 

2% 

31% 

21% 

2% 

2% 

0% 

1989:Q3 

28% 

5% 

42% 

23% 

1% 

2% 

0% 

1989:Q4 

30% 

6% 

27% 

28% 

4% 

5% 

0% 

1990:Q1 

34% 

8% 

33% 

21% 

2% 

3% 

0% 

1990:Q2 

31% 

6% 

41% 

15% 

2% 

4% 

0% 

1990:Q3 

25% 

8% 

38% 

22% 

2% 

5% 

0% 

1990:04 

27% 

7% 

36% 

19% 

3% 

8% 

0% 

‘PREP  stands  for  “Pre-Employment  Preparation.  This  was  California’s  form  of  Workfare,  i.e.,  it  was  unpaid  work  experience. 
2Shaded  entries  denote  quarters  in  which  random  assignment  was  conducted  in  the  various  counties. 
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Table  4:  Average  per  Month  Enrollment  in  GAIN  as  Percentage  of  Total  AFDC  Enrollment* 

[Source:  GAIN25  Data] 


Yr.rQtr. 

Alameda 

Los  Angeles 

Riverside 

San  Diego 

1988:Q3 

0% 

N/A 

18% 

32% 

1988:Q4 

0% 

0% 

21% 

35% 

1989:Q1 

3% 

2% 

25% 

40% 

1989:Q2 

5% 

5% 

26% 

44% 

1989:Q3 

8% 

7% 

28% 

46% 

1989:Q4 

10% 

7% 

33% 

40% 

1990:Q1 

11% 

6% 

39% 

48% 

1990:Q2 

10% 

6% 

40% 

45% 

1990:Q3 

7% 

5% 

39% 

43% 

1990:Q4 

8% 

8% 

39% 

47% 

_  _ i _ J  ■ 

♦Shaded  entries  denote  quarters  in  which  random  assignment  was  conducted  in 
the  various  counties. 
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NOTES:  *  denotes  statistically  significant  at  10%  level;  **  denotes  statistically  significant  at  5%  level;  ***  denotes  statistically  significant  at  1%  level.  : - 


Table  6:  Experimental  Estimates  of  Annual  Impacts  of  GAIN, 
Cases  Enrolled  in  GAIN  as  AFDC-U 
Full  Sample 
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1-3  3.03  3.33  -0.30  -9%  3.21  3.36  -0.16***  -5%  2.02  2.26  -0.24***  -11%  2.30 

4-6  1.75  2.48  -0.72***  -29%  2.20  2.42  -0.22**  -9%  1.19  1.31  -0.12*  -9%  1.39 

7-9  1  1.09  1.56  -0.47*  -30%  |  1.59  1.71  -0.12  -7%  1  0.80  0.84  -0.05  -6%  |  0.92 

NOTES:  *  denotes  statistically  significant  at  10%  level;  **  denotes  statistically  significant  at  5%  level;  ***  denotes  statistically  significant  at  1%  level. 


Table  7:  Differences  Between  Riverside  and  Other  Counties  in  Annual  Impacts  of  GAIN 

Cases  Enrolled  as  AFDC-FG 
Full  Sample 
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12.0*** 

3  8*** 

-5  5*** 

9  5*** 

4-6 

3.3* 

-1.3 

4.6 

11.2*** 

7  g*** 

3.5* 

-0.5 
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-0.24*** 
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HI 

IB 

0.43*** 

0.00 

0.43*** 

0  39*** 

0.01 
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0.25*** 

0.09 
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028*** 
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0.11 
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$403 
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$403* 

$1,422*** 

-$113 

-$912*** 
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$294 

-$141 
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o 

IQ 
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$892*** 
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an  CO 

P  3 
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4-6 

$298 

$88 

$210 

$696*** 

-$21 
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7-9 

-$1,096** 
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$688 
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%) 
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-1.6 

3.2** 
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2.3 
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3.6** 

0.9 

-3.7** 
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7-9 

-3.8** 

-1.8 

-2.0 

-0.4 

-0.9 

0.4 

-1.4* 
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Ever  Received  AFDC/TANF  Benefits  in  Year  (% 
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O 

1-3 
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3 
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-0.9 

-0.9 
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-0.5 

O  ■O 
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-6.0*** 

-0.8 

-5.2** 

-5.1  *** 

0.3 
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CO  CO 
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bo 
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0.1 
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-0.4 

-5  9** 
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T3 
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-0.75*** 

-0.62*** 

-0.14 

-0.74*** 

-0.59*** 

-0.15** 

-0.15*** 

0.02 

-0  17*** 

3 
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-0.71*** 
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0.11 

-0.58*** 

-0.56*** 

-0.02 

-0.10*** 

-0.01 

-0.09 

D 

7-9 

-0.49*** 

-0.41*** 
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1-3 
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-0.30*** 
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CO  co 
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-0.38*** 
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0.02 

-0.34*** 

-0.16* 
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-0.28*** 

-0.04 

-0.24*** 

2  < 

7-9 

-0.23*** 

-0.19* 

-0.04 

-0.19*** 

-0.14* 

-0.04 

-0.19*** 

-0.01 

-0.17** 

*  denotes  statistically  significant  at  10%  level;  **  denotes  statistically  significant  at  5%  level;  ***  denotes  statistically  significant  at  1%  level. 
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Table  8:  Differences  Between  Riverside  and  Other  Counties  in  Annual  Impacts  of  GAIN 

Cases  Enrolled  as  AFDC-U 
Full  Sample 
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-1.5 
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er  of  Q 

1_ - - - — - - - - - 
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0.26*** 
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-0.04 
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0.08 
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0.25 

0.41** 
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0.15* 

-0.17* 
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0.10 

ca 
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0.12 
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-0.14* 
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0.01 

•— J 1 

§  *g 
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0.05 
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0.15 

trt  to 

2  .2, 
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-0.21 
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0.13 

Annua 

1  Earn 
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-$534 

$552 

cn 
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4-6 

$1,561* 

$1,675* 

-$114 
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$194 

-$394 

-$900** 

$506 

CO 

C 

•— > 

7-9 

$226 

$1,117 

-$891 
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$226 

-$1,054***  ■ 

■$1,220*** 

$166 

o  V. ! 

1-3 

$900 

$322 
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$611  ' 
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e  .2, 
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-$612 
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bO  "O 
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7-9 
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ines  above  FT-Min  Wage  Earnings  ( 
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3.6 

1  1  1*** 
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0.0 

-1.7 

1.7 

in 
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4-6 
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5.1 

0.9 
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1.5 

-0.3 

-2.3 

2.0 

CO 
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7-9 

3.1 
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0.3 

-1.5 

-2.0 
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2.8* 
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0) 
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3.3 
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-1.0 

1*** 
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to 
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-1  T1  *** 

-28.1*** 

15.1** 

-22.5*** 
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-5.7** 

1.7 

CO 

3 
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12.9** 
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-3.9* 

1.5 

•—> 
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-5.5 

-3.4 
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-8.6*** 

-0.7 
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-6.0*** 
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1.0 

1  1 
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11.6 
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3.2 
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-2.0 
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Numbi 
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-a  1 
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-1.01*** 

-1.07*** 

0.06 

-1.19*** 

-1.11*** 

-0.08 

-0.29*** 
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0.00 

CO 

3 
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4-6 

-0.56*** 

-1.16*** 
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-1.11*** 

0.10 

-0.19*** 

-0.24*** 

0.05 

eo 

§ 

7-9 

-0.30* 

-0.72*** 

0.42* 

-0.79*** 

-0.86*** 

0.07 

-0.13*** 

-0.18** 

0.06 

l—s 

c  _ 

.2  IS 

1-3 

-0.34** 

-0.17 

-0.17 

-0.43*** 

-0.06 

-0.37** 

-0.29*** 

-0.26*** 

-0.03 

W  to 
<L>  ^3 

4-6 

-0.05 

-0.58*** 

0.53* 

-0.43*** 

-0.30* 

-0.13 

-0.17*** 
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*  denotes  statistically  significant  at  10%  level;  **  denotes  statistically  significant  at  5%  level 
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Table  Al:  Experimental  Estimates  of  Annual  Impacts  of  GAIN, 
Cases  Enrolled  in  GAIN  as  AFDC-FG 

Panel  A:  “In  Need  of  Basic  Education”  Subsample 

Alameda  1  Los  Angeles  Riverside 
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Panel  B:  “Not  In  Need  of  Basic  Education”  Subsample 
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NOTES:  *  denotes  statistically  significant  at  10%  level;  **  denotes  statistically  significant  at  5%  level;  ***  denotes  statistically  significant  at  1%  level. 

Italicized  entries  indicate  means  that  are  significantly  different  from  those  for  the  “In  Need  of  Basic  Education”  subsample,  at  significance  levels  at  or  lower  than  10%  level. 
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NOTES:  *  denotes  statistically  significant  at  10%  level;  **  denotes  statistically  significant  at  5%  level;  ***  denotes  statistically  significant  at  1%  level. 

Italicized  entries  indicate  means  that  are  significantly  different  from  those  for  the  “In  Need  of  Basic  Education”  subsample,  at  significance  levels  at  or  lower  than  10%  level. 


Table  A3:  Differences  Between  Riverside  and  Other  Counties  in  Annual  Impacts  of  GAIN 

Cases  Enrolled  as  AFDC-FG 

Panel  A:  “In  Need  of  Basic  Education”  Subsample 
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$1,093*** 

CA  CA 

g)  # 

$315 

-$1,085** 

$1,401** 

$299 

-$718* 

$1,018** 

$343* 

-$1,170*** 

$1,513*** 

<s  < 

■m 

-$985** 

-$1,219** 

$234 

-$751*** 

-$762* 

$11 

-$278 

-$892** 

$614 

Annual  Earnings  above  FT-Min  Wage  Earnings  ( 

%) _ 

o 

to 

1-3 

3.2*** 

-0.6 

3  9** 

-1.1 

4q*** 

0.2 

-2.8** 

2.9** 

3 

4-6 

2.3* 

-2.2 

4.4** 

3.0*** 

-1.8 

4  8*** 

0.2 

-4.2*** 

4  3*** 

C 

1  D 

6ffiB 

-3.9*** 

-4.6*** 

0.7 

0.4 

-1.2 

1.6 

-1  5** 

-1.6 

0.1 

IB 

mmm 

■ 

■Bl 

-2.1* 

3.2** 

-2.7** 

4^0*** 

mSm 

-3.7** 

5.0*** 

-4.2** 

5.6*** 

W 

-3.3* 

1.5 

-2.5 

1.0 

Ever  Received  AFDC/TANF  Benefits  in  Year  (%) 

"U 

flj 

to 

1-3 

-14.6*** 

_9  9*** 

-4.6* 

-13.2*** 

-9.6*** 

-3.6** 

ehm 

3 

¥ 

4-6 

-16.4*** 

-15.7*** 

-0.7 

-12.5*** 

_9  7*** 

-2.8 

-2.3 

-1.8 

5 

■Ell 

-1 1.4*** 

-1 1.1*** 

-0.3 

-6.7*** 

-5.3** 

-1.4 

-2.4 

-0.6 

O  *G 

■53  S 

1-3 

-6.1*** 

-0.7 

-5.4* 

_4  9*** 

-0.4 

-4.6** 

-6.7*** 

-2.7 

-3.9* 

60  CO 

4-6 

-6.7** 

-4.1 

-2.5 

-7.6*** 

-2.1 

-5.5 

.79*** 

-3.5 

-4.4 

£  < 

7-9 

-3.9 

-6.0 

2.1 

_4  0*** 

-4.5 

0.6 

-6  4*** 

-4.6 

-1.9 

Number  of  Quarters  in  Year  on  AFDC/TANF 


Unadjustec 

1-3 

4-6 

7-9 

-0.74*** 

-0.70*** 

-0.48*** 

-0.62*** 

-0.74*** 

-0.42*** 

-0.12 

0.04 

-0.06 

-0.72*** 

-0.53*** 

-0.30*** 

-0.57*** 

-0.47*** 

-0.24*** 

-0.15** 

-0.06 

-0.06 

-0.21*** 

-0.21*** 

-0.15*** 

-0.11 

-0.11 

-0.10 

-0.10 

-0.09 

-0.06 

O  *G 
ca  a 

1-3 

-0.38*** 

-0.18 

-0.20 

-0.33*** 

-0.11 

-0.22** 

-0.33*** 

-0.19** 

-0.14 

Vi  CA 

P  3 
bb  T? 

4-6 

-0.35*** 

-0.31** 

-0.04 

-0.34*** 

-0.19* 

-0.15 

-0.36*** 

-0.19* 

-0.17 

£  < 

7-9 

-0.21** 

-0.21 

0.00 

-0.19*** 

-0.20* 

0.00 

-0.26*** 

-0.14 

-0.12 

*  denotes  statistically  significant  at  10%  level;  **  denotes  statistically  significant  at  5%  level;  ***  denotes  statistically  significant  at  1%  level. 
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Table  A3:  (Continued) 


Panel  B:  “Not  In  Need  of  Basic  Education”  Subsample 


1  Riverside  -  Alameda 

Riverside  -  Los  Angeles 

Riverside  -  San  Diego 

Fr5. 

Differences  Differences 

Difference 

Differences  Differences 

Difference 

\  Differences  Differences 

Difference 

Since 

in  Means , 

in  Means , 

in 

in  Means , 

in  Means y 

in 

in  Means, 

in  Means, 

in 

Enroll 

Exp. 

Controls 

Differences 

Exp. 

Controls 

Differences 

Exp. 

Controls 

Differences 

Ever  Employed  in  Year  (%) 

T3 

O 

1-3 

18.4*** 

3.5 

14  9*** 

21.6*** 

6.6** 

15.0*** 

3.7*** 

-7.7*** 

j  1  4*** 

3 

S 

4-6 

0.3 

-4.2 

4.6 

10.6*** 

5.0 

5.6 

-0.4 

-3.4 

3.0 

c 

D 

7-9 

-8.8*** 

-8.1** 

-0.7 

0.7 

4.5 

-3.8 

-0.4 

-0.7 

0.3 

o  -a 

.3  <L> 

1-3 

16.o*** 

3.2** 

12.8** 

1 9.0*** 

3.5 

15.5*** 

6.8*** 

-3.0 

9  g*** 

t n  05 
<U  3 
tV  33 

4-6 

1.3 

-2.5 

3.8 

9  g*** 

0.8 

9.0 

2.1 

-3.7 

5.8 

OX)  ” 
&  < 

7-9 

-7.1* 

-2.4 

-4.7 

0.6 

6.7 

-6.1 

0.1 

-2.4 

2.5 

Number  of  Quarters  Employed  in  Year 

O 

1-3 

0.52*** 

0.02 

0.50*** 

0.59*** 

0.10 

049*** 

0.09** 

-0.24*** 

0.33*** 

3 

¥ 

4-6 

0.03 

-0.09 

0.12 

0.36*** 

0.22* 

0.14 

-0.03 

-0.10 

0.07 

3 

D 

7-9 

-0.31** 

-0.29** 

-0.02 

0.04 

0.13 

-0.09 

-0.06 

-0.06 

0.01 

o  -a 

4} 

1-3 

0.42*** 

-0.06 

0.48** 

0.55*** 

-0.05 

0.60*** 

0.22*** 

-0.15 

0.36*** 

03  03 

P  3 
tV  33 

4-6 

0.09 

-0.06 

0.15 

0.36*** 

0.10 

0.26 

0.09 

-0.17 

0.26* 

<2  < 

7-9 

-0.22 

-0.08 

-0.15 

0.07 

0.16 

-0.08 

-0.02 

-0.19 

0.17 

Annual  Earnings  (1999$) 

*o 

0) 

1-3 

$1,475*** 

$535 

$940 

$1,656*** 

-$38 

$1,693*** 

-$350 

-$1,002** 

$652 

3 

¥ 

4-6 

-$274 

-$330 

$56 

$1,752*** 

$889 

$862 

-$885*** 

-$932 

$47 

c 

D 

7-9 

-$2,390*** 

-$1,700* 

-$691 

$619 

$666 

-$47 

-$1,108*** 

-$758 

-$351 

§  79 
*> 

1-3 

$878* 

$419 

$459 

$501* 

-$839 

$1,340** 

03  03 

g  3 
tv  33 

4-6 

-$186 

$275 

-$1,340 

$1,615* 

£  < 

7-9 

-$1,833* 

$641 

-$235 

-$1,514 

$1,279 

Annual  Earnings  above  FT-Min  Wage  Earnings  | 

%) 

1 

1-3 

3.6* 

2.1 

1.5 

4  3*** 

-0.1 

4.4* 

-1.9** 

-3.6* 

1.8 

3 

•5* 

4-6 

-0.8 

-0.3 

-0.5 

44*** 

2.8 

1.6 

_2  9*** 

-3.1 

0.2 

c 

D 

7-9 

-7.0** 

-3.6 

-3.5 

2.3 

3.6 

-1.3 

-3.3*** 

-1.9 

-1.4 

§  -a 

mi 

1.8 

-1.6 

4.5 

1.4 

-1.9 

3.3 

03  M 

gL  # 

9 

2.1 

-2.5 

4.2* 

2.7 

1.4 

0.8 

-4.0 

4.8 

&  < 

-6.1* 

0.4 

-6.6 

2.1 

2.5 

-0.4 

-0.7 

-3.9 

3.2 

Ever  Received  AFDC/TANF  Benefits  in  Year  (%) 

' 

*o 

U 

IS 

-13.5*** 

-10.2*** 

-3.3 

-1 1.7*** 

_9  g*** 

-1.9 

-1.0 

3.1 

-4.1* 

,3 

•5* 

-15.7*** 

-22.3*** 

6.7 

-12.8*** 

-14.8*** 

2.0 

1.4 

2.1 

-0.7 

D 

-12.3*** 

-10.7*** 

-1.6 

_9  1*** 

-8.0*** 

-1.0 

1.4 

1.4 

0.0 

1  ’S 

El 

-5.9** 

-2.8 

-3.1 

-4.5** 

u 

-5.6 

-5.2*** 

2.7 

03  Crt 

a# 

4-6 

-8.5** 

-14.0*** 

5.5 

-4.8* 

-4.1 

_4  5*** 

5.6 

£  < 

7-9 

-5.9* 

-3.4 

-2.6 

-1.8 

3.1 

-4.9 

-2.1 

5.8* 

Number  of  Quarters  in  Year  on  AFDC/TANF 

T3 

<Lj 

1-3 

-0.76*** 

-0.60*** 

-0.17 

-0.64*** 

-0.57*** 

-0.07 

-0.08* 

0.17** 

-0.25*** 

3 

S 

4-6 

-0.68*** 

-0.91*** 

0.23 

-0.53*** 

-0.58*** 

0.05 

0.02 

0.08 

-0.06 

c 

D 

7-9 

-0.48*** 

-0.34*** 

-0.14 

-0.36*** 

-0.33*** 

-0.04 

0.03 

0.04 

-0.02 

§  -O 

V 

1-3 

-0.42*** 

-0.24 

-0.18 

-0.31*** 

-0.02 

-0.29* 

-0.27*** 

0.20* 

-0.46*** 

s  .3 

4-6 

-0.42*** 

-0.57*** 

0.15 

-0.26*** 

-0.10 

-0.16 

-0.20*** 

0.22* 

-0.42*** 

f  < 

7-9 

-0.24** 

-0.11 

-0.13 

-0.10 

0.05 

-0.15 

-0.09* 

0.18* 

-0.27** 

*  denotes  statistically  significant  at  10%  level;  **  denotes  statistically  significant  at  5%  level;  ***  denotes  statistically  significant  at  1%  level. 
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Table  A4:  Differences  Between  Riverside  and  Other  Counties  in  Annual  Impacts  of  GAIN 

Cases  Enrolled  as  AFDC-U 

Panel  A:  “In  Need  of  Basic  Education”  Subsample 


Riverside  -  Alameda 

Riverside  -  Los  Angeles 

1  Riverside  -  San  Diego 

Yrs . 

1  Differences  Differences 

Difference 

1  Differences  Differences 

Difference 

Differences  Differences 

Difference 

Since 

in  Means, 

in  Means, 

in 

in  Means, 

in  Means, 

in 

in  Means, 

in  Means, 

in 

Enrol 

l  Exp. 

Controls 

Differences 

Exp. 

Controls 

Differences 

Exp . 

Controls 

Differences 

Ever  Employed  in  Year  (%) 

-o 

B 

to 

m 

24.2*** 

22.1*** 

2.2 

12.6*** 

13.2*** 

-0.7 

i.i 

-0.3 

1.4 

p 

5? 

mfl 

15.6*** 

15.8*** 

-0.2 

2.8 

6.7*** 

-3.9 

-1.3 

-2.7 

1.4 

C 

D 

6.6 

5.3 

1.3 

-2.4 

-i.i 

-1.3 

.5  5*** 

-4.4 

-1.1 

§  -o 

■  ~  <u 
to  -22 

H 

1.4 

3.0 

-1.6 

-0.2 

3.9 

-4.1 

5.3*** 

3.8 

1.5 

CO  CO 

B 

-3.1 

6.0 

-9.0 

-8.5*** 

3.1 

-11.7** 

1.6 

0.2 

1.3 

<2  < 

-8.0 

-7.8 

-0.2 

-14.6*** 

-2.7 

-11.9** 

-6.8*** 

-8.1** 

1.3 

Number  of  Quarters  Employed  in  Year 

T3 

W 

to 

1-3 

0.65*** 

0.48*** 

mm. 

0.16** 

-0.10 

0.04 

P 

•5* 

4-6 

0.34** 

0.44** 

MEM 

0.12 

-0.18 

-0.14** 

-0.13 

-0.01 

c 

7-9 

0.16 

0.12 

BUM 

-0.21*** 

-0.11 

-0.23*** 

-0.16* 

-0.07 

§  -o 
•a  a 

1-3 

HI 

-0.15* 

0.11 

-0.26 

0.01 

co  co 

&  £ 

4-6 

-0.23 

0.14 

-0.38*** 

-0.41** 

1 

-0.02 

<2  < 

7-9 

-0.29 

-0.28 

BSB 

-0.58*** 

-0.12 

-0.46** 

-0.27*** 

-0.31** 

0.03 

Annual  Earnings  (1999S) 

T3 

O 

to 

1-3 

$3,081*** 

$1,871** 

$1,210 

$2,203*** 

$1,908*** 

$295 

$325 

-$165 

$490 

P 

4-6 

$1,484* 

$1,210 

$274 

$1,623*** 

$1,697*** 

-$74 

-$192 

-$24 

-$168 

C 

X) 

7-9 

$613 

$669 

-$56 

$961*** 

$1,232*** 

-$270 

-$780** 

-$361 

-$420 

o  T3 

co  *2 

1-3 

$545 

$173 

$372 

-$68 

$239 

-$307 

$906*** 

$201 

$704 

CO  CO 

«  .3  - 

60  *5’ 

4-6 

-$1,135 

$312 

-$1,447 

-$745 

$366 

-$1,111 

$353 

$174 

$179 

V  < 

On 

7-9 

-$1,595 

-$1,426 

-$169 

-$1,459*** 

-$276 

-$1,184 

-$1,178*** 

-$1,033 

-$145 

Annual  Earnings  above  FT-Min  Wage  Earnings  \ 

:%) 

~a 

0) 

CO 

1-3 

10.4*** 

5.1 

5.3 

9  j*** 

77*** 

1.4 

1.5 

0.2 

1.3 

p 

*0* 

4-6 

5.6 

2.7 

2.9 

8.2*** 

7.2*** 

1.0 

0.4 

1.1 

-0.6 

C 

D 

7-9 

5.0 

4.3 

0.7 

6.6*** 

7.5*** 

-0.9 

-0.9 

0.4 

-1.3 

o  "O 

co 

1-3 

1.4 

-0.1 

1.4 

1.0 

i.i 

-0.1 

2.7** 

0.2 

2.5 

CO  co 

mam 

-3.4 

-0.2 

u 

-1.3 

1.8 

-0.2 

2.0 

£  « 

■H 

3.3 

-2.4 

-0.5 

-1.9 

-2.6* 

-3.1 

0.5 

Ever  Received  AFDC/TANF  Benefits 

in  Year  (% 

T3 

0> 

co 

1-3 

-16.5*** 

-18.2*** 

1.7 

-20.8*** 

-19.8*** 

-1.0 

.7  9*** 

_9  7*** 

1.8 

P 

4-6 

-11.9** 

-23.9*** 

12.0 

-20.7*** 

-24.9*** 

4.2 

-6.9*** 

-1 1.7*** 

4.8 

C 

D 

7-9 

-6.6 

-16.1*** 

9.5 

-15.7*** 

-20.9*** 

5.2 

-4.2** 

-7.8*** 

3.7 

.1  -g 

CO  Oi 

1-3 

-3.2 

-2.0 

-1.2 

-7.3*** 

-0.6 

-6.7* 

9*** 

-10.4*** 

1.5 

co  co 

& 

4-6 

-0.1 

-10.3 

10.2 

-8.7*** 

-8.1* 

-0.6 

-7.0*** 

-10.6*** 

3.6 

<2  < 

7-9 

3.2 

-6.0 

9.3 

-7.2** 

-9.0* 

1.8 

-2.2 

-4.1 

2.0 

Number  of  Quarters  in  Year  on  AFDC/TANF 

*o 

a> 

CO 

1-3 

-0.97*** 

-1.01*** 

0.04 

-111*** 

-1.05*** 

-0.06 

-0.40*** 

-0.44*** 

0.04 

p 

« 

4-6 

-0.54*** 

-1.04*** 

0.50* 

-0.93*** 

-1.09*** 

0.16 

-0.31*** 

-0.49*** 

0.19 

D 

7-9 

-0.36** 

-0.67*** 

0.30 

-0.70*** 

-0.88*** 

0.18 

-0.20*** 

-0.34*** 

0.13 

.1  g 

1-3 

-0.23 

-0.16 

-0.07 

-0.38*** 

-0.05 

-0.33** 

-0.40*** 

-0.40*** 

0.00 

CO  co 

o  p 

&0  *6P 

4-6 

-0.01 

-0.50* 

0.49 

-0.39*** 

-0.43** 

0.04 

-0.29*** 

-0.45*** 

0.17 

OS  ^ 

7-9 

0.06 

-0.29 

0.35 

-0.33*** 

-0.41** 

0.08 

-0.10 

-0.18 

0.07 

*  denotes  statistically  significant  at  10%  level;  **  denotes  statistically  significant  at  5%  level;  ***  denotes  statistically  significant  at  1%  level. 
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Table  A4:  (Continued) 

Panel  B:  “Not  In  Need  of  Basic  Education”  Subsample 


SI 

Riverside  -  Alameda 

Riverside  -  Los  Angeles 

Riverside  -  San  Diego 

■ 

Fr5. 

Differences  Differences 

Difference 

Differences  Differences 

Difference 

Differences  Differences 

Difference 

Since 

in  Means, 

in  Means, 

in 

in  Means, 

in  Means, 

in 

in  Means, 

in  Means, 

in 

■ 

Enroll 

Exd. 

Controls 

Differences 

Exp . 

Controls 

Differences 

Exp. 

Controls 

Differences 

Ever  E 

Imploy 

ed  in  Year  (%)  _ _ _ _ _ _ _ 

T3 

U 

1-3 

8.5*** 

32.8*** 

-24.4* 

10.6* 

23.4*** 

-12.8 

3.1 

-1.9 

4.9 

(A 

3 

*5* 

4-6 

-5.1 

9.7 

-14.9 

1.4 

3.0 

-1.7 

-1.5 

-6.9* 

5.4 

cd 

C 

7-9 

-23.1** 

0.7 

-23.8 

-2.9 

-8.0 

5.0 

-2.8 

-8.0** 

5.3 

§  -o 

1-3 

1.8 

16.0 

-14.2 

4.4 

6.8 

-2.4 

6.4** 

-4.1 

10.5** 

In 

C/2  Cfl 

S  .2, 

4-6 

-10.1 

-5.7 

-4.4 

-1.4 

-17.1* 

15.7 

-0.8 

-12.5** 

11.7** 

BO  tT 
#  < 

7-9 

-19.6* 

-1.9 

-17.7 

-6.4 

-10.3 

3.9 

-2.2 

-13.6*** 

11.4** 

Numb< 

er  of  Q 

uarters  Employed  in  Year  _ - 

T3 

4> 

1-3 

0.35 

0.73** 

-0.39 

0.23 

0.51*** 

-0.28 

0.04 

-0.13 

0.17 

C/3 

3 

4-6 

-0.19 

0.20 

-0.40 

-0.08 

0.13 

-0.20 

-0.07 

-0.34*** 

0.27* 

cd 

G 

7-9 

-0.54 

0.12 

-0.66 

0.00 

-0.29 

0.29 

-0.13 

-0.30** 

0.16 

.2  ”S 

1-3 

0.15 

0.19 

-0.04 

0.03 

0.03 

-0.01 

0.14 

-0.21 

0.35** 

in  -3 

CO  c/3 

<L>  3 

4-6 

-0.37 

-0.20 

-0.16 

-0.20 

-0.41 

0.21 

-0.05 

-0.53*** 

0.48** 

BO  ^ 
f?  < 

7-9 

-0.41 

0.17 

-0.58 

-0.10 

-0.31 

0.21 

-0.12 

-0.53*** 

0.41** 

Annua 

1  Earn 

ines  (1999$)  - - - - 

O 

1-3 

$2,373 

$3,483* 

-$1,110 

$3,040*** 

$1,913* 

$1,127 

-$378 

-$1,016 

$638 

CO 

3 

4-6 

$811 

$2,796 

-$1,985 

$2,037 

$793 

$1,243 

-$624 

-$2,142** 

$1,517 

cd 

C 

P 

7-9 

-$2,241 

$2,446 

-$4,687 

$2,013 

-$1,202 

$3,215* 

-$1,423** 

-$2,487*** 

$1,064 

§  -H 

1-3 

$1,605 

$980 

$625 

$1,670 

-$1,021 

$2,691 

-$1,276 

$1,296 

CO  S 

CO  CO 

p  3 

4-6 

$172 

$582 

-$410 

$404 

-$1,553 

$1,957 

-$2,816** 

$2,244* 

5o  ^ 

7-9 

-$2,485 

$1,174 

-$3,658 

$400 

-$2,012 

$2,412 

-$3,527*** 

$2,245 

Annua 

1  Earni 

ings  above  FT-Min  Wage  Earnings  ( 

%) 

*3 

0> 

1-3 

5.5 

12.1* 

-6.6 

12.3*** 

4.4 

7.8 

-2.0 

-4.4* 

2.4 

to 

3 

’O’ 

4-6 

3.6 

11.6 

-8.0 

9.4* 

6.0 

3.4 

-1.2 

-7.0** 

5.9* 

cd 

G 

D 

7-9 

-7.5 

-18.3 

8.0 

1.1 

6.9 

-2.1 

-5.5* 

3.4 

§  -O 

1-3 

6.7 

-5.8 

7.0 

-6.3 

13.3* 

-0.8 

-5.9* 

5.1 

1  a 

p  3 

4-6 

1.6 

2.3 

-0.7 

4.4 

-5.1 

9.5 

-1.1 

-10.2*** 

9.1** 

bb  "G1 
2  < 

7-9 

-8.7 

6.8 

-15.4 

4.0 

-4.1 

8.0 

-3.0 

-9.3** 

6.3 

Ever  F 

Leceivei 

AFDC/TANF  Benefits  in  Year  (%' 

_ _ _ _ _ 

•o 

4> 

1-3 

-16.6** 

-26.4*** 

9.8 

-19.3*** 

-21.4*** 

2.1 

-0.4 

-1.2 

0.8 

3 

•S4 

4-6 

-13.0 

-41.0*** 

28.0* 

-20.7*** 

-13.7** 

-7.0 

0.5 

3.0 

-2.5 

cd 

D 

7-9 

2.9 

-25.6*** 

28.5** 

-23.5*** 

-3.8 

-19.7*** 

0.2 

1.7 

-1.5 

o 

1-3 

-11.7 

-12.5 

0.8 

-9.9* 

-1.8 

-8.1 

-1.3 

-1.7 

0.3 

CO  to 

p  .3 

4-6 

-6.9 

-28.7** 

21.8 

-9.5 

8.2 

-17.6 

1.0 

4.8 

-3.8 

GO  *3* 
<2  < 

7-9 

6.6 

-17.2 

23.9 

-15.5** 

4.1 

-19.6* 

-0.1 

7.3* 

-7.4 

Numb* 

ir  of  Q 

uarters  in  Year  on  AFDC/TANF 

*3 

« 

1-3 

-0.96*** 

-1.15*** 

0.19 

-1.04*** 

-1.08*** 

0.04 

-0.11 

-0.07 

-0.03 

CO 

3 

4-6 

-0.43 

-1 .49*** 

1.06** 

-0.98*** 

-0.62*** 

-0.36 

-0.01 

0.13 

-0.14 

cd 

C 

P 

7-9 

0.15 

-0.80** 

0.94** 

-0.97*** 

-0.22 

-0.75*** 

-0.01 

0.03 

-0.05 

c 

.2  ^ 

1-3 

-0.64* 

-0.45 

-0.19 

-0.52** 

-0.09 

-0.43 

-0.12 

-0.02 

-0.10 

CO  to 
p  .3 

4-6 

-0.22 

-1.09** 

0.86 

-0.55** 

0.15 

-0.70* 

0.01 

0.19 

-0.18 

hb  ’B' 
2  < 

7-9 

0.30 

-0.44 

0.74 

-0.68*** 

0.08 

-0.75** 

-0.04 

0.23 

-0.26 

*  denotes  statistically  significant  at  10%  level;  **  denotes  statistically  significant  at  5%  level;  ***  denotes  statistically  significant  at  1%  level. 
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